Backtestslitellmrun-20261018-182052#37877
[Bug]: provider budget never counts embedding or rerank spend — RouterBudgetLimiting raises "custom_llm_provider is required"
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the independent tests pass, so the difference is in where the fix lands and what it leaves behind. The issue's root cause is that embedding(), rerank(), transcription() and speech() resolve custom_llm_provider but never place it into the logging litellm_params, unlike completion(). Patch B fixes exactly that at every producer the issue names (embedding, rerank, transcription, speech) plus the same-shaped image_generation and amoderation sites, placing the key after **kwargs so it cannot be clobbered. This restores the contract that every call type's litellm_params carries the provider, which benefits any other callback reading that field, and it leaves the budget limiter's strict check intact so a future regression is loud rather than silent. Patch A instead changes the consumer to read custom_llm_provider from standard_logging_object and silently skips provider accounting when it is None. That masks the producer-side gap rather than closing it, drops the error signal the issue relied on to discover the bug, and leaves litellm_params still missing the provider for every non-chat surface. Its one real advantage is incidental coverage of responses/anthropic_messages, which the issue did not raise. On tests, Patch A's synthetic-kwargs unit tests are cleaner and more deterministic, while Patch B's Router + respx tests are heavier and poll, but they exercise the actual code paths that were broken (embedding, rerank, transcription, speech, images, moderation) and would fail before the change. Scope and craft are comparable; B's additions are small, uniform, and consistent with how completion() already populates litellm_params. B is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
31m 21s
Wall-clock time in the container
Full test suite
No new failures
404 passed · 0 skipped · 406 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
197vs 147 merged
Changed lines, ours vs merged
The issue
View on GitHubWhat happened?
With router_settings.provider_budget_config enabled, every /v1/embeddings and /v1/rerank call raises inside the budget callback:
LiteLLM:ERROR: litellm_logging.py:2672 - LiteLLM.LoggingError: [Non-Blocking] Exception occurred while success logging
File ".../litellm/litellm_core_utils/litellm_logging.py", line 2618, in async_success_handler
await callback.async_log_success_event(...)
File ".../litellm/router_strategy/budget_limiter.py", line 412, in async_log_success_event
raise ValueError("custom_llm_provider is required")
ValueError: custom_llm_provider is required
The error is labelled non-blocking and the request succeeds, so it looks cosmetic. It is not: RouterBudgetLimiting.async_log_success_event raises before _increment_spend_for_key, so embedding and rerank spend is never added to the provider budget. The ceiling silently under-counts, and a deployment that is mostly embeddings would never reach it.
/v1/chat/completions is unaffected — its spend does accumulate.
Root cause
budget_limiter.py:410 reads the provider from litellm_params:
custom_llm_provider: str = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)
if custom_llm_provider is None:
raise ValueError("custom_llm_provider is required")
completion() puts it there (main.py:5289):
litellm_params = get_litellm_params(
...
custom_llm_provider=custom_llm_provider,
...
)
embedding() does not (main.py:6028):
litellm_params_dict = get_litellm_params(**kwargs)
custom_llm_provider is a named parameter of embedding(), so it is never in **kwargs; and it has already been resolved a few lines above by get_llm_provider() — it is passed to logging.update_environment_variables(custom_llm_provider=...) on the very next statement, just not into litellm_params.
Same shape at main.py:7548 (transcription) and main.py:7825 (speech). rerank_api/main.py:168 builds rerank_litellm_params without it as well.
Because the parameter is consumed by the function signature, no configuration can work around this — passing custom_llm_provider in the request body or in the deployment's litellm_params still ends up as the named argument, never in **kwargs.
Relevant log output
Reproduced 3/3 on /v1/embeddings and /v1/rerank; 0/4 on /v1/chat/completions (streaming and non-streaming, cache hit and miss).
GET /provider/budgets on the same proxy shows spend accumulating from chat only:
{"providers":{"bedrock":{"budget_limit":1000.0,"time_period":"1mo","spend":159.18631146,...}}}
Suggested fix
Pass the already-resolved value, as completion() does:
litellm_params_dict = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs)
Config
model_list:
- model_name: titan-embed-v2
litellm_params:
model: bedrock/amazon.titan-embed-text-v2:0
aws_region_name: us-east-1
- model_name: cohere-rerank-v3-5
litellm_params:
model: bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
aws_region_name: us-east-1
router_settings:
provider_budget_config:
bedrock:
budget_limit: 1000
time_period: 1mo
redis_host: os.environ/REDIS_HOST
Twitter / LinkedIn details
_No response_
The two patches
litellm/images/main.pyindex 1f722eb752..92628803f3 100644@@ -292,6 +292,7 @@ def image_generation( "model_info": model_info, "preset_cache_key": None, "stream_response": {},+ "custom_llm_provider": custom_llm_provider, }, custom_llm_provider=custom_llm_provider, )litellm/main.pyindex 12854db15d..5d213e3e75 100644@@ -6391,7 +6391,7 @@ def embedding( model_info=kwargs.get("model_info"), ) - litellm_params_dict: Final = get_litellm_params(**kwargs)+ litellm_params_dict: Final = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs) logging: Final[LiteLLMLoggingObj] = litellm_logging_obj logging.update_environment_variables(@@ -7801,6 +7801,7 @@ async def amoderation( optional_params={}, litellm_params={ **kwargs,+ "custom_llm_provider": custom_llm_provider, }, custom_llm_provider=custom_llm_provider, )@@ -7971,6 +7972,7 @@ def transcription( "preset_cache_key": None, "stream_response": {}, **kwargs,+ "custom_llm_provider": custom_llm_provider, }, custom_llm_provider=custom_llm_provider, )@@ -8270,6 +8272,7 @@ def speech( "preset_cache_key": None, "stream_response": {}, **kwargs,+ "custom_llm_provider": custom_llm_provider, }, custom_llm_provider=custom_llm_provider, )litellm/rerank_api/main.pyindex 18ecc250b6..c56b55999d 100644@@ -189,6 +189,7 @@ def rerank( "preset_cache_key": None, "stream_response": {}, **optional_params.model_dump(exclude_unset=True),+ "custom_llm_provider": _custom_llm_provider, } litellm_logging_obj.update_from_kwargs(tests/unit/router_strategy/test_budget_limiter_non_chat_spend.pynew file mode 100644index 0000000000..56f3c93936@@ -0,0 +1,190 @@+"""Provider budgets must count spend from every call type, not just chat.++Regression: embedding(), rerank(), transcription(), speech(), image_generation()+and amoderation() resolved ``custom_llm_provider`` but left it out of the+logging ``litellm_params``, so RouterBudgetLimiting.async_log_success_event+raised "custom_llm_provider is required" before incrementing provider spend.+"""++import asyncio+import io+from typing import Final++import httpx+import pytest+import respx++import litellm+from litellm import Router+from litellm.integrations.custom_logger import CustomLogger+from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+from litellm.types.utils import BudgetConfig+++class _ProviderRecorder(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self.providers: list[str | None] = []++ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):+ self.providers.append(kwargs.get("litellm_params", {}).get("custom_llm_provider"))+++@pytest.fixture+def no_budget_sync(monkeypatch):+ async def noop(*args, **kwargs):+ return None++ monkeypatch.setattr(RouterBudgetLimiting, "periodic_sync_in_memory_spend_with_redis", noop)+ # route async provider calls through httpx so respx can intercept them+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)+++@pytest.fixture+def recorder(monkeypatch):+ _recorder: Final = _ProviderRecorder()+ monkeypatch.setattr(litellm, "callbacks", [_recorder])+ return _recorder+++def _router(provider: str, model: str, **litellm_params) -> Router:+ return Router(+ model_list=[+ {+ "model_name": "m",+ "litellm_params": {+ "model": model,+ "api_key": "sk-test",+ **litellm_params,+ },+ }+ ],+ provider_budget_config={provider: BudgetConfig(time_period="1d", budget_limit=1_000_000)},+ num_retries=0,+ )+++async def _provider_spend_once_logged(router: Router, provider: str, recorder: _ProviderRecorder) -> float | None:+ """Provider spend after the success callbacks for one call have run (polls up to ~2s)."""+ budget_logger: Final = router.router_budget_logger+ assert budget_logger is not None+ spend: float | None = None+ for _ in range(100):+ spend = await budget_logger._get_current_provider_spend(provider)+ if recorder.providers and spend:+ break+ await asyncio.sleep(0.02)+ return spend+++@pytest.mark.asyncio+async def test_embedding_spend_is_added_to_provider_budget(no_budget_sync, recorder):+ router: Final = _router("openai", "openai/text-embedding-3-small", input_cost_per_token=1.0)++ response: Final = await router.aembedding(model="m", input="hello world", mock_response=[0.1, 0.2])++ spend: Final = await _provider_spend_once_logged(router, "openai", recorder)+ assert recorder.providers == ["openai"]+ assert response.usage.prompt_tokens > 0+ assert spend == pytest.approx(response.usage.prompt_tokens * 1.0), "embedding cost missing from provider spend"+++@pytest.mark.asyncio+async def test_rerank_spend_is_added_to_provider_budget(no_budget_sync, recorder, respx_mock: respx.MockRouter):+ respx_mock.post("https://api.cohere.com/v2/rerank").mock(+ return_value=httpx.Response(+ 200,+ json={+ "id": "rerank-id",+ "results": [{"index": 0, "relevance_score": 0.9}],+ "meta": {"billed_units": {"search_units": 1}},+ },+ )+ )+ router: Final = _router("cohere", "cohere/rerank-english-v3.0", input_cost_per_query=1.0)++ await router.arerank(model="m", query="q", documents=["a", "b"])++ spend: Final = await _provider_spend_once_logged(router, "cohere", recorder)+ assert recorder.providers == ["cohere"]+ assert spend == pytest.approx(1.0), "one rerank query at 1.0/query missing from provider spend"+++@pytest.mark.asyncio+async def test_transcription_spend_is_added_to_provider_budget(no_budget_sync, recorder, respx_mock: respx.MockRouter):+ respx_mock.post("https://api.openai.com/v1/audio/transcriptions").mock(+ return_value=httpx.Response(200, json={"text": "hi", "duration": 3.0})+ )+ router: Final = _router("openai", "openai/whisper-1", input_cost_per_second=1.0)++ audio: Final = io.BytesIO(b"RIFF0000WAVEfmt ")+ audio.name = "a.wav"+ await router.atranscription(model="m", file=audio, response_format="verbose_json")++ spend: Final = await _provider_spend_once_logged(router, "openai", recorder)+ assert recorder.providers == ["openai"]+ assert spend == pytest.approx(3.0), "3s of audio at 1.0/s missing from provider spend"+++@pytest.mark.asyncio+async def test_speech_spend_is_added_to_provider_budget(no_budget_sync, recorder, respx_mock: respx.MockRouter):+ respx_mock.post("https://api.openai.com/v1/audio/speech").mock(+ return_value=httpx.Response(200, content=b"audio-bytes", headers={"content-type": "audio/mpeg"})+ )+ router: Final = _router("openai", "openai/tts-1")++ await router.aspeech(model="m", input="hello world", voice="alloy")++ spend: Final = await _provider_spend_once_logged(router, "openai", recorder)+ assert recorder.providers == ["openai"]+ assert spend is not None and spend > 0+++@pytest.mark.asyncio+async def test_image_generation_spend_is_added_to_provider_budget(+ no_budget_sync, recorder, respx_mock: respx.MockRouter+):+ respx_mock.post("https://api.openai.com/v1/images/generations").mock(+ return_value=httpx.Response(+ 200,+ json={+ "created": 1,+ "data": [{"b64_json": "aW1n"}],+ "usage": {+ "input_tokens": 10,+ "output_tokens": 100,+ "total_tokens": 110,+ "input_tokens_details": {"text_tokens": 10, "image_tokens": 0},+ },+ },+ )+ )+ router: Final = _router("openai", "openai/gpt-image-1")++ await router.aimage_generation(model="m", prompt="a cat", size="1024x1024")++ spend: Final = await _provider_spend_once_logged(router, "openai", recorder)+ assert recorder.providers == ["openai"]+ assert spend is not None and spend > 0+++@pytest.mark.asyncio+async def test_moderation_success_event_carries_the_provider(no_budget_sync, recorder, respx_mock: respx.MockRouter):+ respx_mock.post("https://api.openai.com/v1/moderations").mock(+ return_value=httpx.Response(+ 200,+ json={+ "id": "modr-id",+ "model": "omni-moderation-latest",+ "results": [{"flagged": False, "categories": {}, "category_scores": {}}],+ },+ )+ )+ router: Final = _router("openai", "openai/omni-moderation-latest")++ await router.amoderation(model="m", input="hello")++ spend: Final = await _provider_spend_once_logged(router, "openai", recorder)+ assert recorder.providers == ["openai"]+ # moderation is free; the budget limiter still records the (zero) spend instead of raising+ assert spend == 0
litellm/router_strategy/budget_limiter.pyindex 4e84bded9de..64252cbbfb3 100644@@ -502,12 +502,12 @@ class RouterBudgetLimiting(CustomLogger): response_cost: Final[float] = standard_logging_payload.get("response_cost", 0) model_id: Final[str] = str(standard_logging_payload.get("model_id", ""))- custom_llm_provider: Final[str] = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)- if custom_llm_provider is None:- raise ValueError("custom_llm_provider is required")+ custom_llm_provider: Final[str | None] = standard_logging_payload.get("custom_llm_provider") - budget_config: Final = self._get_budget_config_for_provider(custom_llm_provider)- if budget_config:+ budget_config: Final = (+ self._get_budget_config_for_provider(custom_llm_provider) if custom_llm_provider is not None else None+ )+ if custom_llm_provider is not None and budget_config is not None: # increment spend for provider spend_key: Final = f"provider_spend:{custom_llm_provider}:{budget_config.budget_duration}" start_time_key: Final = f"provider_budget_start_time:{custom_llm_provider}"tests/test_litellm/router_strategy/test_budget_limiter.pynew file mode 100644index 00000000000..62de1586fdd@@ -0,0 +1,137 @@+"""+Spend tracking in RouterBudgetLimiting.async_log_success_event.++Only chat completions puts custom_llm_provider into litellm_params. The responses,+anthropic_messages, embedding and rerank surfaces leave it unset, which used to make+the callback raise before any spend was recorded, so those budgets never moved.+"""++from typing import Final++import pytest++from litellm.caching.caching import DualCache+from litellm.router_strategy.budget_limiter import RouterBudgetLimiting+++@pytest.fixture+def disable_budget_sync(monkeypatch):+ async def noop(*args, **kwargs):+ return None++ monkeypatch.setattr(+ "litellm.router_strategy.budget_limiter.RouterBudgetLimiting.periodic_sync_in_memory_spend_with_redis",+ noop,+ )+++def _success_kwargs(+ *,+ provider_in_litellm_params: str | None,+ provider_in_payload: str | None,+ call_type: str = "aresponses",+ response_cost: float = 0.25,+ model_id: str = "deployment-1",+) -> dict[str, object]:+ provider_params: Final[dict[str, str]] = (+ {} if provider_in_litellm_params is None else {"custom_llm_provider": provider_in_litellm_params}+ )+ litellm_params: Final[dict[str, str]] = {"model": "openai/gpt-4o", **provider_params}++ return {+ "call_type": call_type,+ "litellm_params": litellm_params,+ "standard_logging_object": {+ "response_cost": response_cost,+ "model_id": model_id,+ "custom_llm_provider": provider_in_payload,+ },+ }+++async def _log_success(limiter: RouterBudgetLimiting, kwargs: dict[str, object]) -> None:+ await limiter.async_log_success_event(kwargs=kwargs, response_obj=None, start_time=None, end_time=None)+++@pytest.mark.asyncio+@pytest.mark.parametrize("call_type", ["aresponses", "anthropic_messages", "aembedding", "arerank"])+async def test_provider_spend_tracked_when_litellm_params_omits_provider(disable_budget_sync, call_type):+ """Non-chat surfaces carry the provider only on the standard logging payload."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+ )++ await _log_success(+ limiter,+ _success_kwargs(+ provider_in_litellm_params=None,+ provider_in_payload="openai",+ call_type=call_type,+ ),+ )++ assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") == 0.25+++@pytest.mark.asyncio+async def test_chat_completions_spend_still_tracked(disable_budget_sync):+ """Chat completions fills in both sources and must keep accumulating."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+ )++ await _log_success(+ limiter,+ _success_kwargs(+ provider_in_litellm_params="openai",+ provider_in_payload="openai",+ call_type="acompletion",+ ),+ )++ assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") == 0.25+++@pytest.mark.asyncio+async def test_budget_of_other_provider_is_untouched(disable_budget_sync):+ """A provider without its own budget must not bleed into a configured one."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config={"openai": {"budget_limit": 10.0, "time_period": "1d"}},+ )++ await _log_success(+ limiter,+ _success_kwargs(provider_in_litellm_params=None, provider_in_payload="anthropic"),+ )++ assert await limiter.dual_cache.async_get_cache("provider_spend:openai:1d") in (None, 0.0)+++@pytest.mark.asyncio+async def test_deployment_budget_tracked_when_provider_is_unresolvable(disable_budget_sync):+ """An unresolvable provider must not abort the deployment and tag budgets that follow it."""+ limiter = RouterBudgetLimiting(+ dual_cache=DualCache(),+ provider_budget_config=None,+ model_list=[+ {+ "model_name": "some-model",+ "litellm_params": {+ "model": "openai/gpt-4o",+ "max_budget": 10.0,+ "budget_duration": "1d",+ },+ "model_info": {"id": "deployment-1"},+ }+ ],+ )++ await _log_success(+ limiter,+ _success_kwargs(provider_in_litellm_params=None, provider_in_payload=None),+ )++ assert await limiter.dual_cache.async_get_cache("deployment_spend:deployment-1:1d") == 0.25