Backtestslitellmrun-20261018-182052#25889

Issue #25889 Merged fix PR # Run 18 Oct 2026 · run-20261018-182052 BerriAI/litellm

TypeError: '>' not supported between 'NoneType' and 'int' when Ollama returns rate limit error

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches stop the TypeError in the scenarios tested, but they differ in how. Patch B adds a one-line guard at the comparison site: when num_retries is None it simply raises, so an explicit `num_retries=None` silently becomes 'zero retries' and the None still leaks into `_metadata['max_retries']` and any other consumer of the value. It treats the symptom at the one line in the stack trace. Patch A fixes the root cause: every entry point that previously did `kwargs.get('num_retries', self.num_retries)` kept an explicit None instead of falling back to the router default, and A replaces those with a helper that treats None as unset, and also normalises in async_function_with_retries for direct callers. That matches the issue's second suggested fix ('ensure num_retries is always initialized') and preserves the router's configured retry behaviour (the independent tests with router default 1 pass under both, but only A actually retries as configured). A also adds meaningful tests that fail on base and cover acompletion, aanthropic_messages (the proxy route from the trace), aimage_generation, direct async_function_with_retries calls, and the helper itself; B adds no tests. A touches more lines, but the churn is uniform, low-risk, and directly tied to the bug rather than unrelated. A wins on correctness and tests.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

18m 44s

Wall-clock time in the container

Full test suite

No new failures

454 passed · 26 skipped · 480 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

171vs 2 merged

Changed lines, ours vs merged

Bug Description

When Ollama returns a rate limit/session usage limit error, litellm crashes with a TypeError instead of properly handling the error and returning a meaningful message to the caller.

Error Details

The original error from Ollama:

litellm.APIConnectionError: Ollama_chatException - {"error":"you ({username}) have reached your session usage limit, upgrade for higher limits: https://ollama.com/upgrade (ref: 54b63ef7-b3a4-402f-bcc2-bca8a892767c)"}

This triggers a secondary crash in the retry logic:

TypeError: '>' not supported between instances of 'NoneType' and 'int'

Stack Trace

File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5647, in async_function_with_retries
    if num_retries > 0:
    ^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'NoneType' and 'int'

Full traceback:

File "/usr/lib/python3.13/site-packages/litellm/proxy/anthropic_endpoints/endpoints.py", line 53, in anthropic_response
    result = await base_llm_response_processor.base_process_llm_request(
File "/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py", line 992, in base_process_llm_request
    responses = await llm_responses
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5083, in async_wrapper
    return await self._ageneric_api_call_with_fallbacks(
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3779, in _ageneric_api_call_with_fallbacks
    raise e
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3766, in _ageneric_api_call_with_fallbacks
    response = await self.async_function_with_fallbacks(**kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5489, in async_function_with_fallbacks
    return await self.async_function_with_fallbacks_common_utils(
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5446, in async_function_with_fallbacks_common_utils
    raise original_exception
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5480, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5647, in async_function_with_retries
    if num_retries > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

Expected Behavior

The rate limit error from Ollama should be caught and properly propagated as an or similar exception with the original error message, allowing the caller to handle it appropriately (e.g., implement backoff, notify user, etc.).

Actual Behavior

Instead of propagating the original error, litellm crashes with a TypeError because num_retries is None when the retry logic attempts to compare it with 0.

Root Cause

In router.py line 5647, the code assumes num_retries is always an integer:

if num_retries > 0:

But num_retries can be None in certain error paths, causing the TypeError.

Suggested Fix

Add a defensive check before the comparison:

if num_retries is not None and num_retries > 0:

Or ensure num_retries is always initialized to 0 instead of None.

Environment

  • Python: 3.13
  • litellm: installed via pip (version visible in path as site-packages)
  • Provider: Ollama (via litellm proxy)

The two patches

Our patch+162 −9
litellm/router.pyindex 9185e437a3..d7e7cb22c9 100644@@ -2252,7 +2252,7 @@ class Router:         - litellm_trace_id         - metadata         """-        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+        kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)         kwargs.setdefault("litellm_trace_id", str(uuid.uuid4()))         model_group_alias: Optional[str] = None         if self._get_model_from_alias(model=model):@@ -2261,6 +2261,18 @@ class Router:             {"model_group": model, "model_group_alias": model_group_alias}         ) +    def _get_num_retries_from_kwargs(self, kwargs: dict) -> int:+        """+        Return the per-request num_retries, falling back to the router default.++        An explicit `num_retries=None` (e.g. `"num_retries": null` in a proxy request+        body or key/team router_settings) is treated as unset.+        """+        num_retries = kwargs.get("num_retries")+        if num_retries is None:+            return self.num_retries+        return num_retries+     def _set_deployment_num_retries_on_exception(         self, exception: Exception, deployment: dict     ) -> None:@@ -2992,7 +3004,7 @@ class Router:             kwargs["model"] = model             kwargs["prompt"] = prompt             kwargs["original_function"] = self._image_generation-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             kwargs.setdefault("metadata", {}).update({"model_group": model})             response = self.function_with_fallbacks(**kwargs) @@ -3051,7 +3063,7 @@ class Router:             kwargs["model"] = model             kwargs["prompt"] = prompt             kwargs["original_function"] = self._aimage_generation-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs) @@ -3414,7 +3426,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["prompt"] = prompt-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             kwargs.setdefault("metadata", {}).update({"model_group": model})              # pick the one that is available (lowest TPM/RPM)@@ -3558,7 +3570,7 @@ class Router:             kwargs["model"] = model             kwargs["adapter_id"] = adapter_id             kwargs["original_function"] = self._aadapter_completion-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             kwargs.setdefault("metadata", {}).update({"model_group": model})             response = await self.async_function_with_fallbacks(**kwargs) @@ -4184,7 +4196,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["original_function"] = self._acreate_file-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)             response = await self.async_function_with_fallbacks(**kwargs) @@ -4463,7 +4475,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["original_function"] = self._acreate_batch-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             metadata_variable_name = _get_router_metadata_variable_name(                 function_name="_acreate_batch"             )@@ -4702,7 +4714,7 @@ class Router:         try:             kwargs["model"] = model             kwargs["original_function"] = self._acancel_batch-            kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+            kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs)             metadata_variable_name = _get_router_metadata_variable_name(                 function_name="_acancel_batch"             )@@ -5685,7 +5697,8 @@ class Router:             "model_group_retry_policy", self.model_group_retry_policy         )         model_group: Optional[str] = kwargs.get("model")-        num_retries = kwargs.pop("num_retries")+        num_retries = self._get_num_retries_from_kwargs(kwargs)+        kwargs.pop("num_retries", None)          ## ADD MODEL GROUP SIZE TO METADATA - used for model_group_rate_limit_error tracking         _metadata: dict = kwargs.get("litellm_metadata", kwargs.get("metadata")) or {}tests/test_litellm/test_router_num_retries_none.pynew file mode 100644index 0000000000..488aefd353@@ -0,0 +1,140 @@+"""+Test that an explicit `num_retries=None` on a request is treated as "unset".++Previously `kwargs.get("num_retries", self.num_retries)` kept the explicit None,+and the router's retry logic crashed with+`TypeError: '>' not supported between instances of 'NoneType' and 'int'`+instead of surfacing the provider error (e.g. an Ollama session usage limit).+"""++from unittest.mock import AsyncMock, patch++import pytest++import litellm+from litellm import Router++OLLAMA_LIMIT_MESSAGE = (+    "you (user) have reached your session usage limit, upgrade for higher limits"+)+++def _make_rate_limit_error():+    return litellm.RateLimitError(+        message=OLLAMA_LIMIT_MESSAGE,+        llm_provider="ollama_chat",+        model="llama3",+    )+++def _create_router(num_retries: int):+    return Router(+        model_list=[+            {+                "model_name": "test-model",+                "litellm_params": {+                    "model": "ollama_chat/llama3",+                    "api_base": "http://localhost:11434",+                },+            },+        ],+        num_retries=num_retries,+        retry_after=0,+    )+++@pytest.mark.asyncio+@pytest.mark.parametrize(+    "call_router",+    [+        pytest.param(+            lambda router: router.acompletion(+                model="test-model",+                messages=[{"role": "user", "content": "hi"}],+                num_retries=None,+            ),+            id="acompletion",+        ),+        pytest.param(+            lambda router: router.aanthropic_messages(+                model="test-model",+                messages=[{"role": "user", "content": "hi"}],+                max_tokens=10,+                num_retries=None,+            ),+            id="aanthropic_messages",+        ),+        pytest.param(+            lambda router: router.aimage_generation(+                prompt="a cat",+                model="test-model",+                num_retries=None,+            ),+            id="aimage_generation",+        ),+    ],+)+async def test_num_retries_none_falls_back_to_router_default(call_router):+    """+    With num_retries=None on the request, the router default (1) is used: the+    call is retried once and the original rate limit error is raised.+    """+    router = _create_router(num_retries=1)+    mock_make_call = AsyncMock(side_effect=_make_rate_limit_error())++    with patch.object(router, "make_call", mock_make_call):+        with pytest.raises(litellm.RateLimitError) as exc_info:+            await call_router(router)++    assert OLLAMA_LIMIT_MESSAGE in str(exc_info.value)+    assert mock_make_call.call_count == 2+++@pytest.mark.asyncio+async def test_num_retries_none_with_zero_router_default_raises_original_error():+    """+    With router num_retries=0 and num_retries=None on the request, the original+    error is raised without retrying (not a TypeError).+    """+    router = _create_router(num_retries=0)+    mock_make_call = AsyncMock(side_effect=_make_rate_limit_error())++    with patch.object(router, "make_call", mock_make_call):+        with pytest.raises(litellm.RateLimitError) as exc_info:+            await router.acompletion(+                model="test-model",+                messages=[{"role": "user", "content": "hi"}],+                num_retries=None,+            )++    assert OLLAMA_LIMIT_MESSAGE in str(exc_info.value)+    assert mock_make_call.call_count == 1+++@pytest.mark.asyncio+async def test_async_function_with_retries_handles_num_retries_none():+    """+    Direct callers of async_function_with_retries passing num_retries=None get+    the original error, not a TypeError.+    """+    router = _create_router(num_retries=1)+    original_function = AsyncMock(side_effect=_make_rate_limit_error())++    with pytest.raises(litellm.RateLimitError):+        await router.async_function_with_retries(+            model="test-model",+            original_function=original_function,+            num_retries=None,+            messages=[{"role": "user", "content": "hi"}],+            metadata={"model_group": "test-model"},+        )++    assert original_function.call_count == 2+++def test_get_num_retries_from_kwargs():+    router = _create_router(num_retries=3)+    assert router._get_num_retries_from_kwargs({}) == 3+    assert router._get_num_retries_from_kwargs({"num_retries": None}) == 3+    assert router._get_num_retries_from_kwargs({"num_retries": 0}) == 0+    assert router._get_num_retries_from_kwargs({"num_retries": 5}) == 5 
Merged fix · PR #+1 −1
litellm/router.pyindex 9185e437a3a..edaf4dec979 100644@@ -5768,7 +5768,7 @@ class Router:             _metadata["max_retries"] = num_retries              ## LOGGING-            if num_retries > 0:+            if num_retries is not None and num_retries > 0:                 kwargs = self.log_retry(kwargs=kwargs, e=original_exception)             else:                 raise