Backtestslitellmrun-20261018-182052#27363
[Bug]: aembedding missing num_retries kwarg causes zero retries and no failover for embedding model groups
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches add the exact line the issue asks for in `aembedding`, so on the core fix they are equivalent, and the independent tests cannot separate them (all four pass on base, A, and B, suggesting the base's `_update_kwargs_before_fallbacks` already covers the basic failover path and the reported symptom is narrower or environment-specific). Beyond that, Patch A ships no tests at all. Patch B adds the same line to the synchronous `embedding()` path for parity, mirrors the `_acompletion` convention of calling `_set_deployment_num_retries_on_exception` so per-deployment `litellm_params.num_retries` is honoured for embeddings (guarded with `deployment = None` initialisation, matching how `_acompletion` does it), and adds a regression test file covering async/sync failover, request-level `num_retries` precedence, and per-deployment retries. The extra scope is modest, follows patterns already present in the file, and adds little risk; the per-deployment tests in particular would fail before B's change and pass after, which is more than A offers. B is the better fix on tests and craft, with correctness at least equal.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
26m 13s
Wall-clock time in the container
Full test suite
No new failures
546 passed · 26 skipped · 572 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
185vs 1 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
Bug Description
When using the LiteLLM proxy router with multiple deployments of the same embedding model_name across different hosts, failover never occurs when a host is unreachable. The router attempts the selected deployment once, fails, and returns an error — no retries, no failover to other deployments in the model group.
Root Cause
In litellm/router.py, the aembedding() method does not set num_retries in kwargs before calling async_function_with_fallbacks(). Compare aembedding to every other router method (acompletion, acreate_file, atext_completion, etc.) which all include this line:
kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)
aembedding (missing the line — bug):
async def aembedding(self, model, input, is_async=True, **kwargs):
try:
kwargs["model"] = model
kwargs["input"] = input
kwargs["original_function"] = self._aembedding # ✓
# kwargs["num_retries"] = ... # ✗ MISSING
self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
response = await self.async_function_with_fallbacks(**kwargs)
acreate_file (correct pattern — for comparison):
async def acreate_file(self, model, **kwargs):
try:
kwargs["model"] = model
kwargs["original_function"] = self._acreate_file
kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries) # ✓ present
self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
response = await self.async_function_with_fallbacks(**kwargs)
Note: _update_kwargs_before_fallbacks also sets num_retries (line 2234), but it uses kwargs.get("num_retries", self.num_retries) — so if num_retries is already in kwargs it won't be overwritten. The problem is that async_function_with_retries pops num_retries from kwargs (line 5564), so on the retry path the key is gone and _update_kwargs_before_fallbacks may not re-set it correctly. The result is num_retries evaluates to 0 inside the retry loop, causing immediate failure with no retries attempted.
Fix
Add the missing line to aembedding() in router.py:
async def aembedding(self, model, input, is_async=True, **kwargs):
try:
kwargs["model"] = model
kwargs["input"] = input
kwargs["original_function"] = self._aembedding
kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries) # ADD THIS
self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
response = await self.async_function_with_fallbacks(**kwargs)
Steps to Reproduce
- Configure LiteLLM proxy with multiple deployments of the same embedding
model_name:
model_list:
- model_name: mxbai-embed-large
litellm_params:
model: ollama/mxbai-embed-large:latest
api_base: http://host1:11434
- model_name: mxbai-embed-large
litellm_params:
model: ollama/mxbai-embed-large:latest
api_base: http://host2:11434
router_settings:
routing_strategy: simple-shuffle
num_retries: 3
allowed_fails: 0
cooldown_time: 60
- Take
host1offline - Send an embedding request for
mxbai-embed-large - Observe the request fails immediately with no retries —
host2is never tried - Repeat the same test with a chat completion model group — failover works correctly, confirming the bug is specific to
aembedding
Additional Context
This bug was discovered while investigating embedding failover in a multi-host Ollama setup. Chat completion failover was working correctly (after separately fixing the APIConnectionError cooldown bug — see related issue #27362 ). Embedding failover was still broken despite identical config, and code inspection revealed this missing line as the cause.
All other async router methods set num_retries in kwargs before calling async_function_with_fallbacks. The aembedding method appears to have been overlooked when this pattern was established or when retries were added to other methods.
Relevant log output
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.82.6 (also confirmed present in v1.82.1)
Twitter / LinkedIn details
_No response_
The two patches
litellm/router.pyindex 7512ee387d..6e21d75ed1 100644@@ -4310,6 +4310,7 @@ class Router: kwargs["model"] = model kwargs["input"] = input kwargs["original_function"] = self._embedding+ kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries) self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs) response = self.function_with_fallbacks(**kwargs) return response@@ -4318,6 +4319,7 @@ class Router: def _embedding(self, input: Union[str, List], model: str, **kwargs): model_name = None+ deployment = None try: verbose_router_logger.debug( f"Inside embedding()- model: {model}; kwargs: {kwargs}"@@ -4370,6 +4372,9 @@ class Router: ) if model_name is not None: self.fail_calls[model_name] += 1+ # Set per-deployment num_retries on exception for retry logic+ if deployment is not None:+ self._set_deployment_num_retries_on_exception(e, deployment) raise e async def aembedding(@@ -4383,6 +4388,7 @@ class Router: kwargs["model"] = model kwargs["input"] = input kwargs["original_function"] = self._aembedding+ kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries) self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs) response = await self.async_function_with_fallbacks(**kwargs) return response@@ -4399,6 +4405,7 @@ class Router: async def _aembedding(self, input: Union[str, List], model: str, **kwargs): model_name = None+ deployment = None try: verbose_router_logger.debug( f"Inside _aembedding()- model: {model}; kwargs: {kwargs}"@@ -4465,6 +4472,9 @@ class Router: ) if model_name is not None: self.fail_calls[model_name] += 1+ # Set per-deployment num_retries on exception for retry logic+ if deployment is not None:+ self._set_deployment_num_retries_on_exception(e, deployment) raise e #### FILES API ####tests/test_litellm/test_router_embedding_retries.pynew file mode 100644index 0000000000..c130ef7923@@ -0,0 +1,175 @@+"""+Regression tests for retries / failover on Router embedding calls.++GitHub Issue: aembedding missing num_retries kwarg causes zero retries and no+failover for embedding model groups.+"""++import pytest++import litellm+from litellm import Router++UNREACHABLE_API_BASE = "http://127.0.0.1:1"+++def _two_deployment_router(num_retries: int) -> Router:+ return Router(+ model_list=[+ {+ "model_name": "embed-group",+ "litellm_params": {+ "model": "ollama/mxbai-embed-large",+ "api_base": UNREACHABLE_API_BASE,+ },+ "model_info": {"id": "offline-host"},+ },+ {+ "model_name": "embed-group",+ "litellm_params": {+ "model": "openai/text-embedding-3-small",+ "api_key": "fake-key",+ "mock_response": [0.1, 0.2, 0.3],+ },+ "model_info": {"id": "online-host"},+ },+ ],+ num_retries=num_retries,+ allowed_fails=0,+ cooldown_time=60,+ routing_strategy="simple-shuffle",+ )+++def _deployment_by_id(router: Router, model_id: str) -> dict:+ return next(d for d in router.model_list if d["model_info"]["id"] == model_id)+++def _pick_offline_first(router: Router, picked: list):+ """Deterministic selection: offline host on the first attempt, then the online host."""++ def _pick(*args, **kwargs):+ model_id = "offline-host" if len(picked) == 0 else "online-host"+ picked.append(model_id)+ return _deployment_by_id(router, model_id)++ return _pick+++@pytest.mark.asyncio+async def test_aembedding_fails_over_to_other_deployment_when_host_unreachable(+ monkeypatch,+):+ router = _two_deployment_router(num_retries=3)+ picked: list = []+ pick = _pick_offline_first(router, picked)++ async def _async_pick(*args, **kwargs):+ return pick(*args, **kwargs)++ monkeypatch.setattr(router, "async_get_available_deployment", _async_pick)++ response = await router.aembedding(model="embed-group", input=["hello"])++ assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+ assert picked == ["offline-host", "online-host"]+++def test_embedding_fails_over_to_other_deployment_when_host_unreachable(+ monkeypatch,+):+ router = _two_deployment_router(num_retries=3)+ picked: list = []+ monkeypatch.setattr(+ router, "get_available_deployment", _pick_offline_first(router, picked)+ )++ response = router.embedding(model="embed-group", input=["hello"])++ assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+ assert picked == ["offline-host", "online-host"]+++@pytest.mark.asyncio+async def test_aembedding_request_level_num_retries_is_respected(monkeypatch):+ """num_retries passed on the request must not be overridden by the router default."""+ router = _two_deployment_router(num_retries=3)+ picked: list = []+ pick = _pick_offline_first(router, picked)++ async def _async_pick(*args, **kwargs):+ return pick(*args, **kwargs)++ monkeypatch.setattr(router, "async_get_available_deployment", _async_pick)++ with pytest.raises(litellm.APIConnectionError):+ await router.aembedding(model="embed-group", input=["hello"], num_retries=0)++ assert picked == ["offline-host"]+++def _single_deployment_router_with_deployment_retries() -> Router:+ return Router(+ model_list=[+ {+ "model_name": "embed-group",+ "litellm_params": {+ "model": "openai/text-embedding-3-small",+ "api_key": "fake-key",+ "num_retries": 2, # per-deployment setting+ },+ },+ ],+ num_retries=0, # router-level setting would mean no retries+ )+++def _connection_error() -> litellm.APIConnectionError:+ return litellm.APIConnectionError(+ message="host unreachable",+ llm_provider="openai",+ model="text-embedding-3-small",+ )+++@pytest.mark.asyncio+async def test_aembedding_uses_per_deployment_num_retries(monkeypatch):+ """+ Errors raised from the embedding call that carry no num_retries must still be+ retried per the deployment's litellm_params.num_retries (parity with acompletion).+ """+ router = _single_deployment_router_with_deployment_retries()+ original_aembedding = litellm.aembedding+ calls: list = []++ async def _flaky_aembedding(**kwargs):+ calls.append(kwargs["model"])+ if len(calls) == 1:+ raise _connection_error()+ return await original_aembedding(**{**kwargs, "mock_response": [0.5]})++ monkeypatch.setattr(litellm, "aembedding", _flaky_aembedding)++ response = await router.aembedding(model="embed-group", input=["hello"])++ assert response.data[0]["embedding"] == [0.5]+ assert len(calls) == 2+++def test_embedding_uses_per_deployment_num_retries(monkeypatch):+ router = _single_deployment_router_with_deployment_retries()+ original_embedding = litellm.embedding+ calls: list = []++ def _flaky_embedding(**kwargs):+ calls.append(kwargs["model"])+ if len(calls) == 1:+ raise _connection_error()+ return original_embedding(**{**kwargs, "mock_response": [0.5]})++ monkeypatch.setattr(litellm, "embedding", _flaky_embedding)++ response = router.embedding(model="embed-group", input=["hello"])++ assert response.data[0]["embedding"] == [0.5]+ assert len(calls) == 2
litellm/router.pyindex 7512ee387dc..f86cbf77512 100644@@ -4383,6 +4383,7 @@ class Router: kwargs["model"] = model kwargs["input"] = input kwargs["original_function"] = self._aembedding+ kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries) self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs) response = await self.async_function_with_fallbacks(**kwargs) return response