Backtestslitellmrun-20261018-182052#27363

Issue #27363 Merged fix PR # Run 18 Oct 2026 · run-20261018-182052 BerriAI/litellm

[Bug]: aembedding missing num_retries kwarg causes zero retries and no failover for embedding model groups

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches add the exact line the issue asks for in `aembedding`, so on the core fix they are equivalent, and the independent tests cannot separate them (all four pass on base, A, and B, suggesting the base's `_update_kwargs_before_fallbacks` already covers the basic failover path and the reported symptom is narrower or environment-specific). Beyond that, Patch A ships no tests at all. Patch B adds the same line to the synchronous `embedding()` path for parity, mirrors the `_acompletion` convention of calling `_set_deployment_num_retries_on_exception` so per-deployment `litellm_params.num_retries` is honoured for embeddings (guarded with `deployment = None` initialisation, matching how `_acompletion` does it), and adds a regression test file covering async/sync failover, request-level `num_retries` precedence, and per-deployment retries. The extra scope is modest, follows patterns already present in the file, and adds little risk; the per-deployment tests in particular would fail before B's change and pass after, which is more than A offers. B is the better fix on tests and craft, with correctness at least equal.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

26m 13s

Wall-clock time in the container

Full test suite

No new failures

546 passed · 26 skipped · 572 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

185vs 1 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Bug Description

When using the LiteLLM proxy router with multiple deployments of the same embedding model_name across different hosts, failover never occurs when a host is unreachable. The router attempts the selected deployment once, fails, and returns an error — no retries, no failover to other deployments in the model group.

Root Cause

In litellm/router.py, the aembedding() method does not set num_retries in kwargs before calling async_function_with_fallbacks(). Compare aembedding to every other router method (acompletion, acreate_file, atext_completion, etc.) which all include this line:

kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)

aembedding (missing the line — bug):

async def aembedding(self, model, input, is_async=True, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["input"] = input
        kwargs["original_function"] = self._aembedding          # ✓
        # kwargs["num_retries"] = ...                           # ✗ MISSING
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

acreate_file (correct pattern — for comparison):

async def acreate_file(self, model, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["original_function"] = self._acreate_file
        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)  # ✓ present
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

Note: _update_kwargs_before_fallbacks also sets num_retries (line 2234), but it uses kwargs.get("num_retries", self.num_retries) — so if num_retries is already in kwargs it won't be overwritten. The problem is that async_function_with_retries pops num_retries from kwargs (line 5564), so on the retry path the key is gone and _update_kwargs_before_fallbacks may not re-set it correctly. The result is num_retries evaluates to 0 inside the retry loop, causing immediate failure with no retries attempted.

Fix

Add the missing line to aembedding() in router.py:

async def aembedding(self, model, input, is_async=True, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["input"] = input
        kwargs["original_function"] = self._aembedding
        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)  # ADD THIS
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

Steps to Reproduce

  1. Configure LiteLLM proxy with multiple deployments of the same embedding model_name:
model_list:
  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large:latest
      api_base: http://host1:11434

  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large:latest
      api_base: http://host2:11434

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 3
  allowed_fails: 0
  cooldown_time: 60
  1. Take host1 offline
  2. Send an embedding request for mxbai-embed-large
  3. Observe the request fails immediately with no retries — host2 is never tried
  4. Repeat the same test with a chat completion model group — failover works correctly, confirming the bug is specific to aembedding

Additional Context

This bug was discovered while investigating embedding failover in a multi-host Ollama setup. Chat completion failover was working correctly (after separately fixing the APIConnectionError cooldown bug — see related issue #27362 ). Embedding failover was still broken despite identical config, and code inspection revealed this missing line as the cause.

All other async router methods set num_retries in kwargs before calling async_function_with_fallbacks. The aembedding method appears to have been overlooked when this pattern was established or when retries were added to other methods.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.6 (also confirmed present in v1.82.1)

Twitter / LinkedIn details

_No response_

The two patches

Our patch+185 −0
litellm/router.pyindex 7512ee387d..6e21d75ed1 100644@@ -4310,6 +4310,7 @@ class Router:             kwargs["model"] = model             kwargs["input"] = input             kwargs["original_function"] = self._embedding+            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = self.function_with_fallbacks(**kwargs)             return response@@ -4318,6 +4319,7 @@ class Router:      def _embedding(self, input: Union[str, List], model: str, **kwargs):         model_name = None+        deployment = None         try:             verbose_router_logger.debug(                 f"Inside embedding()- model: {model}; kwargs: {kwargs}"@@ -4370,6 +4372,9 @@ class Router:             )             if model_name is not None:                 self.fail_calls[model_name] += 1+            # Set per-deployment num_retries on exception for retry logic+            if deployment is not None:+                self._set_deployment_num_retries_on_exception(e, deployment)             raise e      async def aembedding(@@ -4383,6 +4388,7 @@ class Router:             kwargs["model"] = model             kwargs["input"] = input             kwargs["original_function"] = self._aembedding+            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs)             return response@@ -4399,6 +4405,7 @@ class Router:      async def _aembedding(self, input: Union[str, List], model: str, **kwargs):         model_name = None+        deployment = None         try:             verbose_router_logger.debug(                 f"Inside _aembedding()- model: {model}; kwargs: {kwargs}"@@ -4465,6 +4472,9 @@ class Router:             )             if model_name is not None:                 self.fail_calls[model_name] += 1+            # Set per-deployment num_retries on exception for retry logic+            if deployment is not None:+                self._set_deployment_num_retries_on_exception(e, deployment)             raise e      #### FILES API ####tests/test_litellm/test_router_embedding_retries.pynew file mode 100644index 0000000000..c130ef7923@@ -0,0 +1,175 @@+"""+Regression tests for retries / failover on Router embedding calls.++GitHub Issue: aembedding missing num_retries kwarg causes zero retries and no+failover for embedding model groups.+"""++import pytest++import litellm+from litellm import Router++UNREACHABLE_API_BASE = "http://127.0.0.1:1"+++def _two_deployment_router(num_retries: int) -> Router:+    return Router(+        model_list=[+            {+                "model_name": "embed-group",+                "litellm_params": {+                    "model": "ollama/mxbai-embed-large",+                    "api_base": UNREACHABLE_API_BASE,+                },+                "model_info": {"id": "offline-host"},+            },+            {+                "model_name": "embed-group",+                "litellm_params": {+                    "model": "openai/text-embedding-3-small",+                    "api_key": "fake-key",+                    "mock_response": [0.1, 0.2, 0.3],+                },+                "model_info": {"id": "online-host"},+            },+        ],+        num_retries=num_retries,+        allowed_fails=0,+        cooldown_time=60,+        routing_strategy="simple-shuffle",+    )+++def _deployment_by_id(router: Router, model_id: str) -> dict:+    return next(d for d in router.model_list if d["model_info"]["id"] == model_id)+++def _pick_offline_first(router: Router, picked: list):+    """Deterministic selection: offline host on the first attempt, then the online host."""++    def _pick(*args, **kwargs):+        model_id = "offline-host" if len(picked) == 0 else "online-host"+        picked.append(model_id)+        return _deployment_by_id(router, model_id)++    return _pick+++@pytest.mark.asyncio+async def test_aembedding_fails_over_to_other_deployment_when_host_unreachable(+    monkeypatch,+):+    router = _two_deployment_router(num_retries=3)+    picked: list = []+    pick = _pick_offline_first(router, picked)++    async def _async_pick(*args, **kwargs):+        return pick(*args, **kwargs)++    monkeypatch.setattr(router, "async_get_available_deployment", _async_pick)++    response = await router.aembedding(model="embed-group", input=["hello"])++    assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+    assert picked == ["offline-host", "online-host"]+++def test_embedding_fails_over_to_other_deployment_when_host_unreachable(+    monkeypatch,+):+    router = _two_deployment_router(num_retries=3)+    picked: list = []+    monkeypatch.setattr(+        router, "get_available_deployment", _pick_offline_first(router, picked)+    )++    response = router.embedding(model="embed-group", input=["hello"])++    assert response.data[0]["embedding"] == [0.1, 0.2, 0.3]+    assert picked == ["offline-host", "online-host"]+++@pytest.mark.asyncio+async def test_aembedding_request_level_num_retries_is_respected(monkeypatch):+    """num_retries passed on the request must not be overridden by the router default."""+    router = _two_deployment_router(num_retries=3)+    picked: list = []+    pick = _pick_offline_first(router, picked)++    async def _async_pick(*args, **kwargs):+        return pick(*args, **kwargs)++    monkeypatch.setattr(router, "async_get_available_deployment", _async_pick)++    with pytest.raises(litellm.APIConnectionError):+        await router.aembedding(model="embed-group", input=["hello"], num_retries=0)++    assert picked == ["offline-host"]+++def _single_deployment_router_with_deployment_retries() -> Router:+    return Router(+        model_list=[+            {+                "model_name": "embed-group",+                "litellm_params": {+                    "model": "openai/text-embedding-3-small",+                    "api_key": "fake-key",+                    "num_retries": 2,  # per-deployment setting+                },+            },+        ],+        num_retries=0,  # router-level setting would mean no retries+    )+++def _connection_error() -> litellm.APIConnectionError:+    return litellm.APIConnectionError(+        message="host unreachable",+        llm_provider="openai",+        model="text-embedding-3-small",+    )+++@pytest.mark.asyncio+async def test_aembedding_uses_per_deployment_num_retries(monkeypatch):+    """+    Errors raised from the embedding call that carry no num_retries must still be+    retried per the deployment's litellm_params.num_retries (parity with acompletion).+    """+    router = _single_deployment_router_with_deployment_retries()+    original_aembedding = litellm.aembedding+    calls: list = []++    async def _flaky_aembedding(**kwargs):+        calls.append(kwargs["model"])+        if len(calls) == 1:+            raise _connection_error()+        return await original_aembedding(**{**kwargs, "mock_response": [0.5]})++    monkeypatch.setattr(litellm, "aembedding", _flaky_aembedding)++    response = await router.aembedding(model="embed-group", input=["hello"])++    assert response.data[0]["embedding"] == [0.5]+    assert len(calls) == 2+++def test_embedding_uses_per_deployment_num_retries(monkeypatch):+    router = _single_deployment_router_with_deployment_retries()+    original_embedding = litellm.embedding+    calls: list = []++    def _flaky_embedding(**kwargs):+        calls.append(kwargs["model"])+        if len(calls) == 1:+            raise _connection_error()+        return original_embedding(**{**kwargs, "mock_response": [0.5]})++    monkeypatch.setattr(litellm, "embedding", _flaky_embedding)++    response = router.embedding(model="embed-group", input=["hello"])++    assert response.data[0]["embedding"] == [0.5]+    assert len(calls) == 2 
Merged fix · PR #+1 −0
litellm/router.pyindex 7512ee387dc..f86cbf77512 100644@@ -4383,6 +4383,7 @@ class Router:             kwargs["model"] = model             kwargs["input"] = input             kwargs["original_function"] = self._aembedding+            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs)             return response