Backtestslitellmrun-20261019-033420#25889

Issue #25889 Merged fix PR # Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

TypeError: '>' not supported between 'NoneType' and 'int' when Ollama returns rate limit error

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches stop the TypeError, and the independent tests pass on both. Patch A only guards the comparison at the crash site: when num_retries is None it falls into the `else: raise` branch, so the original provider error propagates, but the router's configured retry count is silently ignored for any request where the body or key/team override supplied num_retries=None, and `_metadata['max_retries']` is still written as None. That masks the symptom rather than addressing why None reaches the retry loop. Patch B fixes the root cause: every entry point used `kwargs.get('num_retries', self.num_retries)`, which does not fall back when the key is present with a None value. The new `_get_num_retries_from_kwargs` helper treats explicit None as unset and is applied consistently across `_update_kwargs_before_fallbacks`, the per-call-type entry points, and `async_function_with_retries` itself (with a safe pop). This preserves the router default retry behaviour and still propagates the original RateLimitError when retries are exhausted. Patch B also adds regression tests that cover the None-propagates-error case, the None-falls-back-to-router-default case with a retry count assertion, a non-completion entry point, and the kwargs normalization including the num_retries=0 edge. The broader diff is mechanical and low risk. Patch A adds no tests. B is the better fix on correctness and tests.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

17m

Wall-clock time in the container

Full test suite

No new failures

452 passed · 26 skipped · 478 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

143vs 2 merged

Changed lines, ours vs merged

Bug Description

When Ollama returns a rate limit/session usage limit error, litellm crashes with a TypeError instead of properly handling the error and returning a meaningful message to the caller.

Error Details

The original error from Ollama:

litellm.APIConnectionError: Ollama_chatException - {"error":"you ({username}) have reached your session usage limit, upgrade for higher limits: https://ollama.com/upgrade (ref: 54b63ef7-b3a4-402f-bcc2-bca8a892767c)"}

This triggers a secondary crash in the retry logic:

TypeError: '>' not supported between instances of 'NoneType' and 'int'

Stack Trace

File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5647, in async_function_with_retries
    if num_retries > 0:
    ^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'NoneType' and 'int'

Full traceback:

File "/usr/lib/python3.13/site-packages/litellm/proxy/anthropic_endpoints/endpoints.py", line 53, in anthropic_response
    result = await base_llm_response_processor.base_process_llm_request(
File "/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py", line 992, in base_process_llm_request
    responses = await llm_responses
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5083, in async_wrapper
    return await self._ageneric_api_call_with_fallbacks(
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3779, in _ageneric_api_call_with_fallbacks
    raise e
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3766, in _ageneric_api_call_with_fallbacks
    response = await self.async_function_with_fallbacks(**kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5489, in async_function_with_fallbacks
    return await self.async_function_with_fallbacks_common_utils(
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5446, in async_function_with_fallbacks_common_utils
    raise original_exception
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5480, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5647, in async_function_with_retries
    if num_retries > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

Expected Behavior

The rate limit error from Ollama should be caught and properly propagated as an or similar exception with the original error message, allowing the caller to handle it appropriately (e.g., implement backoff, notify user, etc.).

Actual Behavior

Instead of propagating the original error, litellm crashes with a TypeError because num_retries is None when the retry logic attempts to compare it with 0.

Root Cause

In router.py line 5647, the code assumes num_retries is always an integer:

if num_retries > 0:

But num_retries can be None in certain error paths, causing the TypeError.

Suggested Fix

Add a defensive check before the comparison:

if num_retries is not None and num_retries > 0:

Or ensure num_retries is always initialized to 0 instead of None.

Environment

  • Python: 3.13
  • litellm: installed via pip (version visible in path as site-packages)
  • Provider: Ollama (via litellm proxy)

The two patches

Our patch+134 −9
litellm/router.pyindex 9185e437a3..b98220529c 100644@@ -2252,7 +2252,7 @@ class Router:         - litellm_trace_id         - metadata         """-        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+        kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)         kwargs.setdefault("litellm_trace_id", str(uuid.uuid4()))         model_group_alias: Optional[str] = None         if self._get_model_from_alias(model=model):@@ -2261,6 +2261,18 @@ class Router:             {"model_group": model, "model_group_alias": model_group_alias}         ) +    def _get_num_retries_from_kwargs(self, kwargs: dict) -> int:+        """+        Get num_retries for this request, falling back to the router default.++        An explicit `num_retries=None` (e.g. from a request body or key/team+        router_settings override) is treated as unset.+        """+        num_retries = kwargs.get("num_retries")+        if num_retries is None:+            return self.num_retries+        return num_retries+     def _set_deployment_num_retries_on_exception(         self, exception: Exception, deployment: dict     ) -> None:@@ -2992,7 +3004,7 @@ class Router:             kwargs["model"] = model             kwargs["prompt"] = prompt             kwargs["original_function"] = self._image_generation-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             kwargs.setdefault("metadata", {}).update({"model_group": model})             response = self.function_with_fallbacks(**kwargs) @@ -3051,7 +3063,7 @@ class Router:             kwargs["model"] = model             kwargs["prompt"] = prompt             kwargs["original_function"] = self._aimage_generation-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs) @@ -3414,7 +3426,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["prompt"] = prompt-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             kwargs.setdefault("metadata", {}).update({"model_group": model})              # pick the one that is available (lowest TPM/RPM)@@ -3558,7 +3570,7 @@ class Router:             kwargs["model"] = model             kwargs["adapter_id"] = adapter_id             kwargs["original_function"] = self._aadapter_completion-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             kwargs.setdefault("metadata", {}).update({"model_group": model})             response = await self.async_function_with_fallbacks(**kwargs) @@ -4184,7 +4196,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["original_function"] = self._acreate_file-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs) @@ -4463,7 +4475,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["original_function"] = self._acreate_batch-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             metadata_variable_name = _get_router_metadata_variable_name(                 function_name="_acreate_batch"             )@@ -4702,7 +4714,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["original_function"] = self._acancel_batch-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             metadata_variable_name = _get_router_metadata_variable_name(                 function_name="_acancel_batch"             )@@ -5685,7 +5697,8 @@ class Router:             "model_group_retry_policy", self.model_group_retry_policy         )         model_group: Optional[str] = kwargs.get("model")-        num_retries = kwargs.pop("num_retries")+        num_retries = self._get_num_retries_from_kwargs(kwargs)+        kwargs.pop("num_retries", None)          ## ADD MODEL GROUP SIZE TO METADATA - used for model_group_rate_limit_error tracking         _metadata: dict = kwargs.get("litellm_metadata", kwargs.get("metadata")) or {}tests/test_litellm/test_router_num_retries_none.pynew file mode 100644index 0000000000..13eabffc47@@ -0,0 +1,112 @@+"""+Regression tests: an explicit `num_retries=None` passed to the Router (e.g. from a+proxy request body or a key/team `router_settings` override) must fall back to the+router default instead of crashing the retry logic with+`TypeError: '>' not supported between instances of 'NoneType' and 'int'`.+"""++from unittest.mock import AsyncMock, patch++import pytest++import litellm+from litellm import Router+++def _create_router(num_retries):+    return Router(+        model_list=[+            {+                "model_name": "test-model",+                "litellm_params": {+                    "model": "openai/gpt-4",+                    "api_key": "fake-key",+                },+            },+        ],+        num_retries=num_retries,+    )+++def _ollama_rate_limit_error():+    return litellm.RateLimitError(+        message='{"error":"you have reached your session usage limit, upgrade for higher limits"}',+        llm_provider="ollama_chat",+        model="ollama_chat/llama3",+    )+++@pytest.mark.asyncio+async def test_acompletion_num_retries_none_raises_original_error():+    """Router.acompletion(num_retries=None) propagates the provider error."""+    router = _create_router(num_retries=0)++    with pytest.raises(litellm.RateLimitError):+        await router.acompletion(+            model="test-model",+            messages=[{"role": "user", "content": "hi"}],+            num_retries=None,+            mock_response="litellm.RateLimitError",+        )+++@pytest.mark.asyncio+async def test_async_function_with_retries_num_retries_none_uses_router_default():+    """num_retries=None falls back to router.num_retries and retries accordingly."""+    router = _create_router(num_retries=2)++    call_count = 0++    async def mock_make_call(*args, **kwargs):+        nonlocal call_count+        call_count += 1+        raise _ollama_rate_limit_error()++    with patch.object(router, "make_call", side_effect=mock_make_call), patch.object(+        router, "_async_get_healthy_deployments", return_value=(["d1"], ["d1"])+    ), patch.object(router, "_time_to_sleep_before_retry", return_value=0), patch.object(+        router, "log_retry", side_effect=lambda kwargs, e: kwargs+    ):+        with pytest.raises(litellm.RateLimitError) as exc_info:+            await router.async_function_with_retries(+                model="test-model",+                messages=[{"role": "user", "content": "hi"}],+                original_function=AsyncMock(),+                metadata={},+                num_retries=None,+            )++    assert "session usage limit" in str(exc_info.value)+    # initial call + 2 retries from the router default+    assert call_count == 3+++@pytest.mark.asyncio+async def test_aimage_generation_num_retries_none_raises_original_error():+    """Per-call-type entrypoints (e.g. aimage_generation) also handle num_retries=None."""+    router = _create_router(num_retries=0)++    async def mock_make_call(*args, **kwargs):+        raise _ollama_rate_limit_error()++    with patch.object(router, "make_call", side_effect=mock_make_call):+        with pytest.raises(litellm.RateLimitError):+            await router.aimage_generation(+                prompt="a cat", model="test-model", num_retries=None+            )+++def test_update_kwargs_before_fallbacks_replaces_none_with_default():+    router = _create_router(num_retries=4)++    kwargs = {"num_retries": None}+    router._update_kwargs_before_fallbacks(model="test-model", kwargs=kwargs)+    assert kwargs["num_retries"] == 4++    kwargs = {"num_retries": 1}+    router._update_kwargs_before_fallbacks(model="test-model", kwargs=kwargs)+    assert kwargs["num_retries"] == 1++    kwargs = {"num_retries": 0}+    router._update_kwargs_before_fallbacks(model="test-model", kwargs=kwargs)+    assert kwargs["num_retries"] == 0 
Merged fix · PR #+1 −1
litellm/router.pyindex 9185e437a3a..edaf4dec979 100644@@ -5768,7 +5768,7 @@ class Router:             _metadata["max_retries"] = num_retries              ## LOGGING-            if num_retries > 0:+            if num_retries is not None and num_retries > 0:                 kwargs = self.log_retry(kwargs=kwargs, e=original_exception)             else:                 raise