Backtestslitellmrun-20261019-033420#37877
[Bug]: provider budget never counts embedding or rerank spend — RouterBudgetLimiting raises "custom_llm_provider is required"
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Patch A fixes the root cause the issue identifies: embedding(), transcription(), speech() and rerank() now put the already-resolved custom_llm_provider into litellm_params, exactly as completion() does and as the issue suggests. It also adds a small generic fallback in Logging.update_environment_variables that copies the provider kwarg into litellm_params when absent, which covers other hand-built surfaces (moderation, etc.) without touching each one, and it preserves an explicit litellm_params value. Its tests exercise the real pipeline: an actual aembedding call with a mock response captured through the callback and then fed into RouterBudgetLimiting, plus handler-patched rerank/transcription/speech runs asserting the provider lands in both the logged and handler litellm_params. Patch B instead changes the consumer to read custom_llm_provider from the standard logging payload and silently skips accounting when it is None. That makes the four independent tests pass, but it leaves litellm_params incomplete for any other callback that reads it, and it converts the loud 'custom_llm_provider is required' error into exactly the silent under-counting the issue warns about. B's tests are synthetic kwargs dicts that never run through embedding/rerank, so they verify the limiter's new branch rather than the reported end-to-end behaviour. A is more correct at the root, better tested, and matches the surrounding conventions; B is a narrower symptom-level change with a masking risk.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
26m 46s
Wall-clock time in the container
Full test suite
No new failures
5,022 passed · 1 skipped · 5,025 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
192vs 147 merged
Changed lines, ours vs merged
The issue
View on GitHubWhat happened?
With router_settings.provider_budget_config enabled, every /v1/embeddings and /v1/rerank call raises inside the budget callback:
LiteLLM:ERROR: litellm_logging.py:2672 - LiteLLM.LoggingError: [Non-Blocking] Exception occurred while success logging
File ".../litellm/litellm_core_utils/litellm_logging.py", line 2618, in async_success_handler
await callback.async_log_success_event(...)
File ".../litellm/router_strategy/budget_limiter.py", line 412, in async_log_success_event
raise ValueError("custom_llm_provider is required")
ValueError: custom_llm_provider is required
The error is labelled non-blocking and the request succeeds, so it looks cosmetic. It is not: RouterBudgetLimiting.async_log_success_event raises before _increment_spend_for_key, so embedding and rerank spend is never added to the provider budget. The ceiling silently under-counts, and a deployment that is mostly embeddings would never reach it.
/v1/chat/completions is unaffected — its spend does accumulate.
Root cause
budget_limiter.py:410 reads the provider from litellm_params:
custom_llm_provider: str = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)
if custom_llm_provider is None:
raise ValueError("custom_llm_provider is required")
completion() puts it there (main.py:5289):
litellm_params = get_litellm_params(
...
custom_llm_provider=custom_llm_provider,
...
)
embedding() does not (main.py:6028):
litellm_params_dict = get_litellm_params(**kwargs)
custom_llm_provider is a named parameter of embedding(), so it is never in **kwargs; and it has already been resolved a few lines above by get_llm_provider() — it is passed to logging.update_environment_variables(custom_llm_provider=...) on the very next statement, just not into litellm_params.
Same shape at main.py:7548 (transcription) and main.py:7825 (speech). rerank_api/main.py:168 builds rerank_litellm_params without it as well.
Because the parameter is consumed by the function signature, no configuration can work around this — passing custom_llm_provider in the request body or in the deployment's litellm_params still ends up as the named argument, never in **kwargs.
Relevant log output
Reproduced 3/3 on /v1/embeddings and /v1/rerank; 0/4 on /v1/chat/completions (streaming and non-streaming, cache hit and miss).
GET /provider/budgets on the same proxy shows spend accumulating from chat only:
{"providers":{"bedrock":{"budget_limit":1000.0,"time_period":"1mo","spend":159.18631146,...}}}
Suggested fix
Pass the already-resolved value, as completion() does:
litellm_params_dict = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs)
Config
model_list:
- model_name: titan-embed-v2
litellm_params:
model: bedrock/amazon.titan-embed-text-v2:0
aws_region_name: us-east-1
- model_name: cohere-rerank-v3-5
litellm_params:
model: bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
aws_region_name: us-east-1
router_settings:
provider_budget_config:
bedrock:
budget_limit: 1000
time_period: 1mo
redis_host: os.environ/REDIS_HOST
Twitter / LinkedIn details
_No response_
The two patches
litellm/litellm_core_utils/litellm_logging.pyindex 83ab2bc11a..78d4b46f58 100644@@ -911,6 +911,11 @@ class Logging(LiteLLMLoggingBaseClass): **self.litellm_params, **scrub_sensitive_keys_in_metadata(litellm_params), }+ # Endpoints that hand-build litellm_params pass the resolved provider only as a kwarg;+ # success callbacks (e.g. provider budgets) read it from litellm_params.+ resolved_custom_llm_provider: Final = additional_params.get("custom_llm_provider")+ if resolved_custom_llm_provider is not None and self.litellm_params.get("custom_llm_provider") is None:+ self.litellm_params["custom_llm_provider"] = resolved_custom_llm_provider self.litellm_request_debug = litellm_params.get("litellm_request_debug", False) self.logger_fn = litellm_params.get("logger_fn", None) if _is_debugging_on() or self.litellm_request_debug:litellm/main.pyindex 12854db15d..ec51c6106d 100644@@ -6391,7 +6391,7 @@ def embedding( model_info=kwargs.get("model_info"), ) - litellm_params_dict: Final = get_litellm_params(**kwargs)+ litellm_params_dict: Final = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs) logging: Final[LiteLLMLoggingObj] = litellm_logging_obj logging.update_environment_variables(@@ -7957,7 +7957,7 @@ def transcription( **non_default_params, ) - litellm_params_dict: Final = get_litellm_params(**kwargs)+ litellm_params_dict: Final = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs) litellm_logging_obj.update_environment_variables( model=model,@@ -8239,7 +8239,12 @@ def speech( if max_retries is None: max_retries = litellm.num_retries or openai.DEFAULT_MAX_RETRIES- litellm_params_dict: Final = get_litellm_params(metadata=metadata, api_key=api_key or dynamic_api_key, **kwargs)+ litellm_params_dict: Final = get_litellm_params(+ metadata=metadata,+ api_key=api_key or dynamic_api_key,+ custom_llm_provider=custom_llm_provider,+ **kwargs,+ ) # Get provider-specific text-to-speech config and map parameters text_to_speech_provider_config = ProviderConfigManager.get_provider_text_to_speech_config(litellm/rerank_api/main.pyindex 18ecc250b6..c56b55999d 100644@@ -189,6 +189,7 @@ def rerank( "preset_cache_key": None, "stream_response": {}, **optional_params.model_dump(exclude_unset=True),+ "custom_llm_provider": _custom_llm_provider, } litellm_logging_obj.update_from_kwargs(tests/test_litellm/router_strategy/test_budget_limiter_non_chat_provider.pynew file mode 100644index 0000000000..0e55e86800@@ -0,0 +1,175 @@+"""+Regression tests: provider budgets must count spend from non-chat endpoints.++embedding / rerank / transcription / speech used to log `litellm_params` without+`custom_llm_provider`, so RouterBudgetLimiting.async_log_success_event raised+"custom_llm_provider is required" before incrementing provider spend.+"""++import asyncio+from typing import Final+from unittest.mock import patch++import pytest++import litellm+from litellm.caching.caching import DualCache+from litellm.integrations.custom_logger import CustomLogger+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj+from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+from litellm.types.utils import BudgetConfig+++class _CaptureSuccessKwargs(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self.kwargs: dict | None = None++ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):+ self.kwargs = kwargs+++class _HandlerReached(Exception):+ pass+++@pytest.fixture+def disable_budget_sync(monkeypatch):+ async def noop(*args, **kwargs):+ return None++ monkeypatch.setattr(+ "litellm.router_strategy.budget_limiter.RouterBudgetLimiting.periodic_sync_in_memory_spend_with_redis",+ noop,+ )+++@pytest.mark.asyncio+async def test_embedding_spend_is_added_to_provider_budget(disable_budget_sync, monkeypatch):+ capture: Final = _CaptureSuccessKwargs()+ monkeypatch.setattr(litellm, "callbacks", [capture])++ await litellm.aembedding(+ model="openai/text-embedding-3-small",+ input=["hello world"],+ mock_response=[0.1, 0.2, 0.3],+ )+ for _ in range(50):+ if capture.kwargs is not None:+ break+ await asyncio.sleep(0.05)+ assert capture.kwargs is not None+ assert capture.kwargs["litellm_params"]["custom_llm_provider"] == "openai"++ provider_budget: Final = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": BudgetConfig(budget_limit=100, time_period="1d")},+ )+ capture.kwargs["standard_logging_object"]["response_cost"] = 0.25+ await provider_budget.async_log_success_event(capture.kwargs, None, None, None)++ spend: Final = await provider_budget.dual_cache.in_memory_cache.async_get_cache("provider_spend:openai:1d")+ assert spend == pytest.approx(0.25)+++def test_rerank_logs_custom_llm_provider_in_litellm_params():+ captured: dict = {}++ def _fake_rerank(*args, **kwargs):+ captured["logged"] = kwargs["logging_obj"].model_call_details["litellm_params"]+ captured["handler"] = kwargs["litellm_params"]+ raise _HandlerReached()++ with patch("litellm.rerank_api.main.base_llm_http_handler.rerank", side_effect=_fake_rerank):+ with pytest.raises(Exception):+ litellm.rerank(+ model="cohere/rerank-english-v3.0",+ query="hello",+ documents=["hello world"],+ api_key="fake-key",+ )++ assert captured["logged"]["custom_llm_provider"] == "cohere"+ assert captured["handler"]["custom_llm_provider"] == "cohere"+++def test_transcription_logs_custom_llm_provider_in_litellm_params():+ captured: dict = {}++ def _fake_transcription(*args, **kwargs):+ captured["logged"] = kwargs["logging_obj"].model_call_details["litellm_params"]+ captured["handler"] = kwargs["litellm_params"]+ raise _HandlerReached()++ with patch("litellm.main.openai_audio_transcriptions.audio_transcriptions", side_effect=_fake_transcription):+ with pytest.raises(Exception):+ litellm.transcription(+ model="openai/whisper-1",+ file=("audio.wav", b"RIFF0000WAVEfmt ", "audio/wav"),+ api_key="fake-key",+ )++ assert captured["logged"]["custom_llm_provider"] == "openai"+ assert captured["handler"]["custom_llm_provider"] == "openai"+++def test_speech_logs_custom_llm_provider_in_litellm_params():+ captured: dict = {}++ def _fake_speech(*args, **kwargs):+ captured["logged"] = kwargs["logging_obj"].model_call_details["litellm_params"]+ raise _HandlerReached()++ with patch("litellm.main.openai_chat_completions.audio_speech", side_effect=_fake_speech):+ with pytest.raises(Exception):+ litellm.speech(+ model="openai/tts-1",+ input="hello",+ voice="alloy",+ api_key="fake-key",+ )++ assert captured["logged"]["custom_llm_provider"] == "openai"+++def test_moderation_logs_custom_llm_provider_in_litellm_params():+ """amoderation hand-builds litellm_params and passes the provider only as a kwarg."""+ logging_obj: Final = LiteLLMLoggingObj(+ model="omni-moderation-latest",+ messages=[],+ stream=False,+ call_type="moderation",+ start_time=None,+ litellm_call_id="test-call-id",+ function_id="test-function-id",+ )+ logging_obj.update_environment_variables(+ model="omni-moderation-latest",+ user=None,+ optional_params={},+ litellm_params={},+ custom_llm_provider="openai",+ )++ assert logging_obj.model_call_details["litellm_params"]["custom_llm_provider"] == "openai"+++def test_explicit_litellm_params_provider_is_not_overridden():+ logging_obj: Final = LiteLLMLoggingObj(+ model="gpt-4o",+ messages=[],+ stream=False,+ call_type="completion",+ start_time=None,+ litellm_call_id="test-call-id",+ function_id="test-function-id",+ )+ logging_obj.update_environment_variables(+ model="gpt-4o",+ user=None,+ optional_params={},+ litellm_params={"custom_llm_provider": "azure"},+ custom_llm_provider="openai",+ )++ assert logging_obj.model_call_details["litellm_params"]["custom_llm_provider"] == "azure"
litellm/router_strategy/budget_limiter.pyindex 4e84bded9de..64252cbbfb3 100644@@ -502,12 +502,12 @@ class RouterBudgetLimiting(CustomLogger): response_cost: Final[float] = standard_logging_payload.get("response_cost", 0) model_id: Final[str] = str(standard_logging_payload.get("model_id", ""))- custom_llm_provider: Final[str] = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)- if custom_llm_provider is None:- raise ValueError("custom_llm_provider is required")+ custom_llm_provider: Final[str | None] = standard_logging_payload.get("custom_llm_provider") - budget_config: Final = self._get_budget_config_for_provider(custom_llm_provider)- if budget_config:+ budget_config: Final = (+ self._get_budget_config_for_provider(custom_llm_provider) if custom_llm_provider is not None else None+ )+ if custom_llm_provider is not None and budget_config is not None: # increment spend for provider spend_key: Final = f"provider_spend:{custom_llm_provider}:{budget_config.budget_duration}" start_time_key: Final = f"provider_budget_start_time:{custom_llm_provider}"tests/test_litellm/router_strategy/test_budget_limiter.pynew file mode 100644index 00000000000..62de1586fdd@@ -0,0 +1,137 @@+"""+Spend tracking in RouterBudgetLimiting.async_log_success_event.++Only chat completions puts custom_llm_provider into litellm_params. The responses,+anthropic_messages, embedding and rerank surfaces leave it unset, which used to make+the callback raise before any spend was recorded, so those budgets never moved.+"""++from typing import Final++import pytest++from litellm.caching.caching import DualCache+from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+++@pytest.fixture+def disable_budget_sync(monkeypatch):+ async def noop(*args, **kwargs):+ return None++ monkeypatch.setattr(+ "litellm.router_strategy.budget_limiter.RouterBudgetLimiting.periodic_sync_in_memory_spend_with_redis",+ noop,+ )+++def _success_kwargs(+ *,+ provider_in_litellm_params: str | None,+ provider_in_payload: str | None,+ call_type: str = "aresponses",+ response_cost: float = 0.25,+ model_id: str = "deployment-1",+) -> dict[str, object]:+ provider_params: Final[dict[str, str]] = (+ {} if provider_in_litellm_params is None else {"custom_llm_provider": provider_in_litellm_params}+ )+ litellm_params: Final[dict[str, str]] = {"model": "openai/gpt-4o", **provider_params}++ return {+ "call_type": call_type,+ "litellm_params": litellm_params,+ "standard_logging_object": {+ "response_cost": response_cost,+ "model_id": model_id,+ "custom_llm_provider": provider_in_payload,+ },+ }+++async def _log_success(limiter: RouterBudgetLimiting, kwargs: dict[str, object]) -> None:+ await limiter.async_log_success_event(kwargs=kwargs, response_obj=None, start_time=None, end_time=None)+++@pytest.mark.asyncio+@pytest.mark.parametrize("call_type", ["aresponses", "anthropic_messages", "aembedding", "arerank"])+async def test_provider_spend_tracked_when_litellm_params_omits_provider(disable_budget_sync, call_type):+ """Non-chat surfaces carry the provider only on the standard logging payload."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+ )++ await _log_success(+ limiter,+ _success_kwargs(+ provider_in_litellm_params=None,+ provider_in_payload="openai",+ call_type=call_type,+ ),+ )++ assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") == 0.25+++@pytest.mark.asyncio+async def test_chat_completions_spend_still_tracked(disable_budget_sync):+ """Chat completions fills in both sources and must keep accumulating."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+ )++ await _log_success(+ limiter,+ _success_kwargs(+ provider_in_litellm_params="openai",+ provider_in_payload="openai",+ call_type="acompletion",+ ),+ )++ assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") == 0.25+++@pytest.mark.asyncio+async def test_budget_of_other_provider_is_untouched(disable_budget_sync):+ """A provider without its own budget must not bleed into a configured one."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+ )++ await _log_success(+ limiter,+ _success_kwargs(provider_in_litellm_params=None, provider_in_payload="anthropic"),+ )++ assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") in (None, 0.0)+++@pytest.mark.asyncio+async def test_deployment_budget_tracked_when_provider_is_unresolvable(disable_budget_sync):+ """An unresolvable provider must not abort the deployment and tag budgets that follow it."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config=None,+ model_list=[+ {+ "model_name": "some-model",+ "litellm_params": {+ "model": "openai/gpt-4o",+ "max_budget": 10.0,+ "budget_duration": "1d",+ },+ "model_info": {"id": "deployment-1"},+ }+ ],+ )++ await _log_success(+ limiter,+ _success_kwargs(provider_in_litellm_params=None, provider_in_payload=None),+ )++ assert await limiter.dual_cache.async_get_cache("deployment_spend:deployment-1:1d") == 0.25