Backtestslitellmrun-20261018-182052#25889
TypeError: '>' not supported between 'NoneType' and 'int' when Ollama returns rate limit error
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches stop the TypeError in the scenarios tested, but they differ in how. Patch B adds a one-line guard at the comparison site: when num_retries is None it simply raises, so an explicit `num_retries=None` silently becomes 'zero retries' and the None still leaks into `_metadata['max_retries']` and any other consumer of the value. It treats the symptom at the one line in the stack trace. Patch A fixes the root cause: every entry point that previously did `kwargs.get('num_retries', self.num_retries)` kept an explicit None instead of falling back to the router default, and A replaces those with a helper that treats None as unset, and also normalises in async_function_with_retries for direct callers. That matches the issue's second suggested fix ('ensure num_retries is always initialized') and preserves the router's configured retry behaviour (the independent tests with router default 1 pass under both, but only A actually retries as configured). A also adds meaningful tests that fail on base and cover acompletion, aanthropic_messages (the proxy route from the trace), aimage_generation, direct async_function_with_retries calls, and the helper itself; B adds no tests. A touches more lines, but the churn is uniform, low-risk, and directly tied to the bug rather than unrelated. A wins on correctness and tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
18m 44s
Wall-clock time in the container
Full test suite
No new failures
454 passed · 26 skipped · 480 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
171vs 2 merged
Changed lines, ours vs merged
The issue
View on GitHubBug Description
When Ollama returns a rate limit/session usage limit error, litellm crashes with a TypeError instead of properly handling the error and returning a meaningful message to the caller.
Error Details
The original error from Ollama:
litellm.APIConnectionError: Ollama_chatException - {"error":"you ({username}) have reached your session usage limit, upgrade for higher limits: https://ollama.com/upgrade (ref: 54b63ef7-b3a4-402f-bcc2-bca8a892767c)"}
This triggers a secondary crash in the retry logic:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
Stack Trace
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5647, in async_function_with_retries
if num_retries > 0:
^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'NoneType' and 'int'
Full traceback:
File "/usr/lib/python3.13/site-packages/litellm/proxy/anthropic_endpoints/endpoints.py", line 53, in anthropic_response
result = await base_llm_response_processor.base_process_llm_request(
File "/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py", line 992, in base_process_llm_request
responses = await llm_responses
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5083, in async_wrapper
return await self._ageneric_api_call_with_fallbacks(
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3779, in _ageneric_api_call_with_fallbacks
raise e
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3766, in _ageneric_api_call_with_fallbacks
response = await self.async_function_with_fallbacks(**kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5489, in async_function_with_fallbacks
return await self.async_function_with_fallbacks_common_utils(
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5446, in async_function_with_fallbacks_common_utils
raise original_exception
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5480, in async_function_with_fallbacks
response = await self.async_function_with_retries(*args, **kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5647, in async_function_with_retries
if num_retries > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
Expected Behavior
The rate limit error from Ollama should be caught and properly propagated as an or similar exception with the original error message, allowing the caller to handle it appropriately (e.g., implement backoff, notify user, etc.).
Actual Behavior
Instead of propagating the original error, litellm crashes with a TypeError because num_retries is None when the retry logic attempts to compare it with 0.
Root Cause
In router.py line 5647, the code assumes num_retries is always an integer:
if num_retries > 0:
But num_retries can be None in certain error paths, causing the TypeError.
Suggested Fix
Add a defensive check before the comparison:
if num_retries is not None and num_retries > 0:
Or ensure num_retries is always initialized to 0 instead of None.
Environment
- Python: 3.13
- litellm: installed via pip (version visible in path as site-packages)
- Provider: Ollama (via litellm proxy)
The two patches
litellm/router.pyindex 9185e437a3..d7e7cb22c9 100644@@ -2252,7 +2252,7 @@ class Router: - litellm_trace_id - metadata """- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) kwargs.setdefault("litellm_trace_id", str(uuid.uuid4())) model_group_alias: Optional[str] = None if self._get_model_from_alias(model=model):@@ -2261,6 +2261,18 @@ class Router: {"model_group": model, "model_group_alias": model_group_alias} ) + def _get_num_retries_from_kwargs(self, kwargs: dict) -> int:+ """+ Return the per-request num_retries, falling back to the router default.++ An explicit `num_retries=None` (e.g. `"num_retries": null` in a proxy request+ body or key/team router_settings) is treated as unset.+ """+ num_retries = kwargs.get("num_retries")+ if num_retries is None:+ return self.num_retries+ return num_retries+ def _set_deployment_num_retries_on_exception( self, exception: Exception, deployment: dict ) -> None:@@ -2992,7 +3004,7 @@ class Router: kwargs["model"] = model kwargs["prompt"] = prompt kwargs["original_function"] = self._image_generation- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) kwargs.setdefault("metadata", {}).update({"model_group": model}) response = self.function_with_fallbacks(**kwargs) @@ -3051,7 +3063,7 @@ class Router: kwargs["model"] = model kwargs["prompt"] = prompt kwargs["original_function"] = self._aimage_generation- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs) response = await self.async_function_with_fallbacks(**kwargs) @@ -3414,7 +3426,7 @@ class Router: try: kwargs["model"] = model kwargs["prompt"] = prompt- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) kwargs.setdefault("metadata", {}).update({"model_group": model}) # pick the one that is available (lowest TPM/RPM)@@ -3558,7 +3570,7 @@ class Router: kwargs["model"] = model kwargs["adapter_id"] = adapter_id kwargs["original_function"] = self._aadapter_completion- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) kwargs.setdefault("metadata", {}).update({"model_group": model}) response = await self.async_function_with_fallbacks(**kwargs) @@ -4184,7 +4196,7 @@ class Router: try: kwargs["model"] = model kwargs["original_function"] = self._acreate_file- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs) response = await self.async_function_with_fallbacks(**kwargs) @@ -4463,7 +4475,7 @@ class Router: try: kwargs["model"] = model kwargs["original_function"] = self._acreate_batch- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) metadata_variable_name = _get_router_metadata_variable_name( function_name="_acreate_batch" )@@ -4702,7 +4714,7 @@ class Router: try: kwargs["model"] = model kwargs["original_function"] = self._acancel_batch- kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)+ kwargs["num_retries"] = self._get_num_retries_from_kwargs(kwargs) metadata_variable_name = _get_router_metadata_variable_name( function_name="_acancel_batch" )@@ -5685,7 +5697,8 @@ class Router: "model_group_retry_policy", self.model_group_retry_policy ) model_group: Optional[str] = kwargs.get("model")- num_retries = kwargs.pop("num_retries")+ num_retries = self._get_num_retries_from_kwargs(kwargs)+ kwargs.pop("num_retries", None) ## ADD MODEL GROUP SIZE TO METADATA - used for model_group_rate_limit_error tracking _metadata: dict = kwargs.get("litellm_metadata", kwargs.get("metadata")) or {}tests/test_litellm/test_router_num_retries_none.pynew file mode 100644index 0000000000..488aefd353@@ -0,0 +1,140 @@+"""+Test that an explicit `num_retries=None` on a request is treated as "unset".++Previously `kwargs.get("num_retries", self.num_retries)` kept the explicit None,+and the router's retry logic crashed with+`TypeError: '>' not supported between instances of 'NoneType' and 'int'`+instead of surfacing the provider error (e.g. an Ollama session usage limit).+"""++from unittest.mock import AsyncMock, patch++import pytest++import litellm+from litellm import Router++OLLAMA_LIMIT_MESSAGE = (+ "you (user) have reached your session usage limit, upgrade for higher limits"+)+++def _make_rate_limit_error():+ return litellm.RateLimitError(+ message=OLLAMA_LIMIT_MESSAGE,+ llm_provider="ollama_chat",+ model="llama3",+ )+++def _create_router(num_retries: int):+ return Router(+ model_list=[+ {+ "model_name": "test-model",+ "litellm_params": {+ "model": "ollama_chat/llama3",+ "api_base": "http://localhost:11434",+ },+ },+ ],+ num_retries=num_retries,+ retry_after=0,+ )+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "call_router",+ [+ pytest.param(+ lambda router: router.acompletion(+ model="test-model",+ messages=[{"role": "user", "content": "hi"}],+ num_retries=None,+ ),+ id="acompletion",+ ),+ pytest.param(+ lambda router: router.aanthropic_messages(+ model="test-model",+ messages=[{"role": "user", "content": "hi"}],+ max_tokens=10,+ num_retries=None,+ ),+ id="aanthropic_messages",+ ),+ pytest.param(+ lambda router: router.aimage_generation(+ prompt="a cat",+ model="test-model",+ num_retries=None,+ ),+ id="aimage_generation",+ ),+ ],+)+async def test_num_retries_none_falls_back_to_router_default(call_router):+ """+ With num_retries=None on the request, the router default (1) is used: the+ call is retried once and the original rate limit error is raised.+ """+ router = _create_router(num_retries=1)+ mock_make_call = AsyncMock(side_effect=_make_rate_limit_error())++ with patch.object(router, "make_call", mock_make_call):+ with pytest.raises(litellm.RateLimitError) as exc_info:+ await call_router(router)++ assert OLLAMA_LIMIT_MESSAGE in str(exc_info.value)+ assert mock_make_call.call_count == 2+++@pytest.mark.asyncio+async def test_num_retries_none_with_zero_router_default_raises_original_error():+ """+ With router num_retries=0 and num_retries=None on the request, the original+ error is raised without retrying (not a TypeError).+ """+ router = _create_router(num_retries=0)+ mock_make_call = AsyncMock(side_effect=_make_rate_limit_error())++ with patch.object(router, "make_call", mock_make_call):+ with pytest.raises(litellm.RateLimitError) as exc_info:+ await router.acompletion(+ model="test-model",+ messages=[{"role": "user", "content": "hi"}],+ num_retries=None,+ )++ assert OLLAMA_LIMIT_MESSAGE in str(exc_info.value)+ assert mock_make_call.call_count == 1+++@pytest.mark.asyncio+async def test_async_function_with_retries_handles_num_retries_none():+ """+ Direct callers of async_function_with_retries passing num_retries=None get+ the original error, not a TypeError.+ """+ router = _create_router(num_retries=1)+ original_function = AsyncMock(side_effect=_make_rate_limit_error())++ with pytest.raises(litellm.RateLimitError):+ await router.async_function_with_retries(+ model="test-model",+ original_function=original_function,+ num_retries=None,+ messages=[{"role": "user", "content": "hi"}],+ metadata={"model_group": "test-model"},+ )++ assert original_function.call_count == 2+++def test_get_num_retries_from_kwargs():+ router = _create_router(num_retries=3)+ assert router._get_num_retries_from_kwargs({}) == 3+ assert router._get_num_retries_from_kwargs({"num_retries": None}) == 3+ assert router._get_num_retries_from_kwargs({"num_retries": 0}) == 0+ assert router._get_num_retries_from_kwargs({"num_retries": 5}) == 5
litellm/router.pyindex 9185e437a3a..edaf4dec979 100644@@ -5768,7 +5768,7 @@ class Router: _metadata["max_retries"] = num_retries ## LOGGING- if num_retries > 0:+ if num_retries is not None and num_retries > 0: kwargs = self.log_retry(kwargs=kwargs, e=original_exception) else: raise