Backtestslitellmrun-20261019-033420#27363

Issue #27363 Merged fix PR # Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: aembedding missing num_retries kwarg causes zero retries and no failover for embedding model groups

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Patch A adds only the single line the issue proposes. But the independent tests show the plain failover scenario already passes on base, meaning `_update_kwargs_before_fallbacks` already sets `num_retries` and the one-liner is effectively a no-op for the reported behaviour. The scenarios that actually fail on base are the ones where `num_retries` arrives as an explicit `None` (the proxy path forwarding key/team router_settings), producing `TypeError: '>' not supported between 'NoneType' and 'int'` deep in the retry loop. Patch A leaves both of those failing, so it does not fix the proxy failover the issue reports. Patch B adds the same line to both `embedding` and `aembedding`, and additionally hardens `async_function_with_retries` so an explicit `num_retries=None` falls back to `self.num_retries`. That resolves the real root cause: all four independent tests pass on B, including the proxy key/team settings scenario. B also ships a regression test file covering aembedding, sync embedding, explicit None, explicit zero (ensuring the None guard does not swallow a deliberate 0), and acompletion with None. The extra `async_function_with_retries` change is small, targeted, and directly tied to the failure mode, so it is not scope creep. B is the clearly better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

25m 52s

Wall-clock time in the container

Full test suite

No new failures

546 passed · 26 skipped · 572 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

149vs 1 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Bug Description

When using the LiteLLM proxy router with multiple deployments of the same embedding model_name across different hosts, failover never occurs when a host is unreachable. The router attempts the selected deployment once, fails, and returns an error — no retries, no failover to other deployments in the model group.

Root Cause

In litellm/router.py, the aembedding() method does not set num_retries in kwargs before calling async_function_with_fallbacks(). Compare aembedding to every other router method (acompletion, acreate_file, atext_completion, etc.) which all include this line:

kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)

aembedding (missing the line — bug):

async def aembedding(self, model, input, is_async=True, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["input"] = input
        kwargs["original_function"] = self._aembedding          # ✓
        # kwargs["num_retries"] = ...                           # ✗ MISSING
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

acreate_file (correct pattern — for comparison):

async def acreate_file(self, model, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["original_function"] = self._acreate_file
        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)  # ✓ present
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

Note: _update_kwargs_before_fallbacks also sets num_retries (line 2234), but it uses kwargs.get("num_retries", self.num_retries) — so if num_retries is already in kwargs it won't be overwritten. The problem is that async_function_with_retries pops num_retries from kwargs (line 5564), so on the retry path the key is gone and _update_kwargs_before_fallbacks may not re-set it correctly. The result is num_retries evaluates to 0 inside the retry loop, causing immediate failure with no retries attempted.

Fix

Add the missing line to aembedding() in router.py:

async def aembedding(self, model, input, is_async=True, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["input"] = input
        kwargs["original_function"] = self._aembedding
        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)  # ADD THIS
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

Steps to Reproduce

  1. Configure LiteLLM proxy with multiple deployments of the same embedding model_name:
model_list:
  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large:latest
      api_base: http://host1:11434

  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large:latest
      api_base: http://host2:11434

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 3
  allowed_fails: 0
  cooldown_time: 60
  1. Take host1 offline
  2. Send an embedding request for mxbai-embed-large
  3. Observe the request fails immediately with no retries — host2 is never tried
  4. Repeat the same test with a chat completion model group — failover works correctly, confirming the bug is specific to aembedding

Additional Context

This bug was discovered while investigating embedding failover in a multi-host Ollama setup. Chat completion failover was working correctly (after separately fixing the APIConnectionError cooldown bug — see related issue #27362 ). Embedding failover was still broken despite identical config, and code inspection revealed this missing line as the cause.

All other async router methods set num_retries in kwargs before calling async_function_with_fallbacks. The aembedding method appears to have been overlooked when this pattern was established or when retries were added to other methods.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.6 (also confirmed present in v1.82.1)

Twitter / LinkedIn details

_No response_

The two patches

Our patch+148 −1
litellm/router.pyindex 7512ee387d..38ab56588a 100644@@ -4310,6 +4310,7 @@ class Router:             kwargs["model"] = model             kwargs["input"] = input             kwargs["original_function"] = self._embedding+            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = self.function_with_fallbacks(**kwargs)             return response@@ -4383,6 +4384,7 @@ class Router:             kwargs["model"] = model             kwargs["input"] = input             kwargs["original_function"] = self._aembedding+            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs)             return response@@ -6007,7 +6009,10 @@ class Router:             "model_group_retry_policy", self.model_group_retry_policy         )         model_group: Optional[str] = kwargs.get("model")-        num_retries = kwargs.pop("num_retries")+        num_retries = kwargs.pop("num_retries", None)+        # an explicit `num_retries=None` (e.g. from key/team router_settings) means "not set"+        if num_retries is None:+            num_retries = self.num_retries          ## ADD MODEL GROUP SIZE TO METADATA - used for model_group_rate_limit_error tracking         _metadata: dict = kwargs.get("litellm_metadata", kwargs.get("metadata")) or {}tests/test_litellm/test_router_embedding_retries.pynew file mode 100644index 0000000000..4c7dccb353@@ -0,0 +1,142 @@+"""+Regression tests: Router embedding calls must retry and fail over to other+deployments in the same model group when one deployment is unreachable.++The first deployment pick is forced onto an unreachable host and retries onto the+healthy one; the request must succeed via the healthy deployment.+"""++import pytest++import litellm+from litellm import Router++MODEL_GROUP = "mxbai-embed-large"+++def _make_router(**router_kwargs) -> Router:+    return Router(+        model_list=[+            {+                "model_name": MODEL_GROUP,+                "litellm_params": {+                    "model": "ollama/mxbai-embed-large:latest",+                    "api_base": "http://127.0.0.1:1",  # unreachable host+                },+                "model_info": {"id": "down-host"},+            },+            {+                "model_name": MODEL_GROUP,+                "litellm_params": {+                    "model": "ollama/mxbai-embed-large:latest",+                    "api_base": "http://127.0.0.1:2",+                    "mock_response": [0.1, 0.2, 0.3],+                },+                "model_info": {"id": "up-host"},+            },+            {+                "model_name": "chat-group",+                "litellm_params": {+                    "model": "ollama/llama3",+                    "api_base": "http://127.0.0.1:1",  # unreachable host+                },+                "model_info": {"id": "chat-down-host"},+            },+            {+                "model_name": "chat-group",+                "litellm_params": {+                    "model": "ollama/llama3",+                    "api_base": "http://127.0.0.1:2",+                    "mock_response": "hello from up host",+                },+                "model_info": {"id": "chat-up-host"},+            },+        ],+        routing_strategy="simple-shuffle",+        allowed_fails=0,+        cooldown_time=60,+        retry_after=0,+        **router_kwargs,+    )+++def _force_first_pick_down_host(+    router: Router, monkeypatch, down_id="down-host", up_id="up-host"+):+    """Route the first attempt to the unreachable host and retries to the healthy one."""+    picks = []++    def _pick(*args, **kwargs):+        model_id = up_id if picks else down_id+        picks.append(model_id)+        return router.get_deployment(model_id=model_id).model_dump(exclude_none=True)++    async def _async_pick(*args, **kwargs):+        return _pick(*args, **kwargs)++    monkeypatch.setattr(router, "async_get_available_deployment", _async_pick)+    monkeypatch.setattr(router, "get_available_deployment", _pick)+    return picks+++@pytest.mark.asyncio+async def test_aembedding_fails_over_when_deployment_unreachable(monkeypatch):+    router = _make_router(num_retries=3)+    picks = _force_first_pick_down_host(router, monkeypatch)++    response = await router.aembedding(model=MODEL_GROUP, input=["hello"])++    assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+    assert picks == ["down-host", "up-host"]+++@pytest.mark.asyncio+async def test_aembedding_explicit_none_num_retries_uses_router_default(monkeypatch):+    """num_retries=None (e.g. unset key/team router_settings) must not disable retries."""+    router = _make_router(num_retries=3)+    picks = _force_first_pick_down_host(router, monkeypatch)++    response = await router.aembedding(+        model=MODEL_GROUP, input=["hello"], num_retries=None+    )++    assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+    assert picks == ["down-host", "up-host"]+++@pytest.mark.asyncio+async def test_aembedding_respects_explicit_zero_num_retries(monkeypatch):+    router = _make_router(num_retries=3)+    picks = _force_first_pick_down_host(router, monkeypatch)++    with pytest.raises(litellm.APIConnectionError):+        await router.aembedding(model=MODEL_GROUP, input=["hello"], num_retries=0)++    assert picks == ["down-host"]+++def test_embedding_fails_over_when_deployment_unreachable(monkeypatch):+    router = _make_router(num_retries=3)+    picks = _force_first_pick_down_host(router, monkeypatch)++    response = router.embedding(model=MODEL_GROUP, input=["hello"], num_retries=None)++    assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+    assert picks == ["down-host", "up-host"]+++@pytest.mark.asyncio+async def test_acompletion_explicit_none_num_retries_uses_router_default(monkeypatch):+    router = _make_router(num_retries=3)+    picks = _force_first_pick_down_host(+        router, monkeypatch, down_id="chat-down-host", up_id="chat-up-host"+    )++    response = await router.acompletion(+        model="chat-group",+        messages=[{"role": "user", "content": "hi"}],+        num_retries=None,+    )++    assert response.choices[0].message.content == "hello from up host"+    assert picks == ["chat-down-host", "chat-up-host"] 
Merged fix · PR #+1 −0
litellm/router.pyindex 7512ee387dc..f86cbf77512 100644@@ -4383,6 +4383,7 @@ class Router:             kwargs["model"] = model             kwargs["input"] = input             kwargs["original_function"] = self._aembedding+            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs)             return response