Backtestslitellmrun-20261019-033420#26939
Title: Vertex AI Gemini streaming silently drops per-request timeout, falling back to litellm.request_timeout (6000s)
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the same core change: add a `timeout` parameter to `make_sync_call` (and the async `make_call`), forward it to `client.post(..., timeout=timeout)`, and pass `timeout=timeout` through both streaming `partial(...)` constructions. The independent tests confirm both resolve the primary bug (all seven fail on base, pass on A and B). Patch B goes further in two ways the issue explicitly asks for. First, it bakes the timeout into the freshly created `HTTPHandler(timeout=timeout)` when no client is supplied, matching the issue's suggested fix; Patch A leaves `HTTPHandler()` with the 5s default, relying solely on the per-call override. Second, B fixes the misleading `Connection timed out after None seconds` message in the sync `HTTPHandler` methods by falling back to `self.client.timeout` when no per-call timeout is given, which the issue called out as the sync handler having been missed. This is a small, low-risk change that addresses a documented symptom rather than unrelated churn. On tests, B's are self-contained unit tests using `httpx.MockTransport` that inspect `request.extensions['timeout']` to prove the per-request timeout actually reaches the wire for both default and user-supplied clients, sync and async, plus a stall test asserting the message names the real timeout and a direct test of the handler message fallback. Patch A's tests depend on `integration._support.wire` and `integration._support.vertex` helpers whose existence cannot be confirmed from the diff, and they do not exercise the `None`-in-message symptom at all. Scope and craft are otherwise comparable; B's `HTTPHandler(timeout=timeout)` is safe since the constructor substitutes its default when given None. B is the more complete and better-verified fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
18m 34s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
155vs 159 merged
Changed lines, ours vs merged
The issue
View on GitHubSummary
When calling litellm.completion(..., stream=True) against a Vertex AI / Google AI Studio Gemini model, the per-request timeout argument is silently discarded. Streams that stall before the first token hang for the full litellm.request_timeout (default 6000s = 100 minutes) instead of failing at the configured timeout. The non-streaming branch of the same handler propagates timeout correctly — only the streaming branch is broken.
Version
litellm == 1.83.0 (also reproduces on main as of filing)
Python 3.10
Provider: vertex_ai and gemini (both routes through the same handler)
Repro
import litellm
import time
Set a small request_timeout so the bug doesn't take 100 minutes to surface.
In practice, the bug means the per-call timeout is ignored regardless.
litellm.request_timeout = 30
start = time.monotonic()
try:
response = litellm.completion(
model="vertex_ai/gemini-2.5-flash",
messages=[{"role": "user", "content": "hello"}],
stream=True,
timeout=2, # expect to fail in ~2s on a stalled stream
)
for chunk in response:
pass
except Exception as e:
print(f"failed after {time.monotonic() - start:.1f}s with: {e!s}")
When the upstream stream stalls (we observe this regularly under load against vertex_ai/gemini-2.5-flash), the call hangs ~30s (the litellm.request_timeout), not ~2s (the per-call timeout). The exception text is also misleading:
litellm.Timeout: Connection timed out after None seconds.
— note the literal None, despite a numeric timeout=2 having been passed.
Root cause
In litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py, VertexLLM.completion() has two branches. The non-streaming branch correctly bakes the request timeout into the httpx client:
non-streaming branch (~line 2971)
if client is None or isinstance(client, AsyncHTTPHandler):
_params = {}
if timeout is not None:
if isinstance(timeout, float) or isinstance(timeout, int):
timeout = httpx.Timeout(timeout)
_params["timeout"] = timeout
client = _get_httpx_client(params=_params)
The streaming branch (~line 2944) constructs the make_sync_call partial without any timeout argument:
SYNC STREAMING CALL
if stream is True:
request_data_str = json.dumps(data)
streaming_response = CustomStreamWrapper(
completion_stream=None,
make_call=partial(
make_sync_call,
gemini_client=(
client
if client is not None and isinstance(client, HTTPHandler)
else None
),
api_base=url,
data=request_data_str,
model=model,
messages=messages,
logging_obj=logging_obj,
headers=headers,
),
...
)
return streaming_response
make_sync_call itself does not accept a timeout parameter, and its client.post(...) invocation does not pass one:
def make_sync_call(
client, gemini_client, api_base, headers, data, model, messages, logging_obj,
):
if gemini_client is not None:
client = gemini_client
if client is None:
client = HTTPHandler() # uses _DEFAULT_TIMEOUT = httpx.Timeout(5.0, connect=5.0)
response = client.post(
api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj,
)
Net result: the user-supplied timeout is dropped on the floor. Whatever httpx.Client instance is reused/created by make_sync_call is what governs the streaming read timeout.
Why "Connection timed out after None seconds"
In litellm/llms/custom_httpx/http_handler.py:1031, the sync HTTPHandler.post() reports timeouts using its local timeout parameter:
except httpx.TimeoutException:
raise litellm.Timeout(
message=f"Connection timed out after {timeout} seconds.",
model="default-model-name",
llm_provider="litellm-httpx-handler",
)
Because make_sync_call calls client.post(...) without a timeout argument, the local timeout is None here even though the actual httpx.Client instance has a real timeout configured. The exception message thus reports None. (The async sibling at line 488 was patched to log Timeout passed={timeout}, time taken={time_delta} — the sync handler was missed.)
Expected behavior
The streaming branch should mirror the non-streaming branch: when timeout is provided to completion(), propagate it down to make_sync_call so that client.post(..., timeout=timeout) enforces the user's configured timeout.
Suggested fix
Two small changes:
Make make_sync_call accept and forward timeout:
def make_sync_call(
client,
gemini_client,
api_base,
headers,
data,
model,
messages,
logging_obj,
timeout: Optional[Union[float, httpx.Timeout]] = None, # NEW
):
if gemini_client is not None:
client = gemini_client
if client is None:
client = HTTPHandler(timeout=timeout) # NEW: bake into fresh client
response = client.post(
api_base,
headers=headers,
data=data,
stream=True,
timeout=timeout, # NEW
logging_obj=logging_obj,
)
...
Pass timeout through the streaming partial(...) in VertexLLM.completion():
make_call=partial(
make_sync_call,
gemini_client=(
client
if client is not None and isinstance(client, HTTPHandler)
else None
),
api_base=url,
data=request_data_str,
model=model,
messages=messages,
logging_obj=logging_obj,
headers=headers,
timeout=timeout, # NEW
),
The async streaming sibling (async_streaming at ~line 2575) already receives timeout via its signature; only the sync streaming path is broken.
Impact
For any sync streaming Gemini call where the upstream stalls before the first chunk:
The user's timeout is ignored.
The call hangs for the full litellm.request_timeout (default 6000s).
The error message reports Connection timed out after None seconds, hiding the actual configured timeout.
In production we observed individual calls hanging 6,008–6,017 seconds — exact matches for litellm.request_timeout=6000 plus connect overhead.
Workaround
Until this is fixed, users of streaming Gemini must rely on application-level timeout managers (e.g., a thread that raises TimeoutError into the main thread) or explicitly construct a short-timeout HTTPHandler and pass it as the client kwarg.
The two patches
litellm/llms/custom_httpx/http_handler.pyindex cc34b4ff87..e17ca53043 100644@@ -1544,7 +1544,7 @@ class HTTPHandler: return response except httpx.TimeoutException: raise litellm.Timeout(- message=f"Connection timed out after {timeout} seconds.",+ message=f"Connection timed out after {timeout if timeout is not None else self.client.timeout} seconds.", model="default-model-name", llm_provider="litellm-httpx-handler", )@@ -1596,7 +1596,7 @@ class HTTPHandler: return response except httpx.TimeoutException: raise litellm.Timeout(- message=f"Connection timed out after {timeout} seconds.",+ message=f"Connection timed out after {timeout if timeout is not None else self.client.timeout} seconds.", model="default-model-name", llm_provider="litellm-httpx-handler", )@@ -1647,7 +1647,7 @@ class HTTPHandler: return response except httpx.TimeoutException: raise litellm.Timeout(- message=f"Connection timed out after {timeout} seconds.",+ message=f"Connection timed out after {timeout if timeout is not None else self.client.timeout} seconds.", model="default-model-name", llm_provider="litellm-httpx-handler", )@@ -1699,7 +1699,7 @@ class HTTPHandler: return response except httpx.TimeoutException: raise litellm.Timeout(- message=f"Connection timed out after {timeout} seconds.",+ message=f"Connection timed out after {timeout if timeout is not None else self.client.timeout} seconds.", model="default-model-name", llm_provider="litellm-httpx-handler", )litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.pyindex 517ac9e98b..46d6832636 100644@@ -2748,6 +2748,7 @@ async def make_call( model: str, messages: list, logging_obj,+ timeout: float | httpx.Timeout | None = None, ): if gemini_client is not None: client = gemini_client@@ -2757,7 +2758,9 @@ async def make_call( ) try:- response: Final = await client.post(api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj)+ response: Final = await client.post(+ api_base, headers=headers, data=data, stream=True, timeout=timeout, logging_obj=logging_obj+ ) response.raise_for_status() except httpx.HTTPStatusError as e: exception_string: Final = str(await e.response.aread())@@ -2800,13 +2803,16 @@ def make_sync_call( model: str, messages: list, logging_obj,+ timeout: float | httpx.Timeout | None = None, ): if gemini_client is not None: client = gemini_client if client is None:- client = HTTPHandler() # Create a new client if none provided+ client = HTTPHandler(timeout=timeout) # Create a new client if none provided - response: Final = client.post(api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj)+ response: Final = client.post(+ api_base, headers=headers, data=data, stream=True, timeout=timeout, logging_obj=logging_obj+ ) if response.status_code != 200 and response.status_code != 201: raise VertexAIError(@@ -2924,6 +2930,7 @@ class VertexLLM(VertexBase): model=model, messages=messages, logging_obj=logging_obj,+ timeout=timeout, ), model=model, custom_llm_provider="vertex_ai_beta",@@ -3200,6 +3207,7 @@ class VertexLLM(VertexBase): messages=messages, logging_obj=logging_obj, headers=headers,+ timeout=timeout, ), model=model, custom_llm_provider="vertex_ai_beta",tests/unit/llms/vertex_ai/gemini/test_gemini_streaming_timeout.pynew file mode 100644index 0000000000..7ce7f9a395@@ -0,0 +1,133 @@+"""+Regression tests: the per-request ``timeout`` passed to ``litellm.completion`` /+``litellm.acompletion`` with ``stream=True`` must reach the HTTP request sent by the+Gemini / Vertex AI streaming handler, instead of silently falling back to the+client's default timeout.+"""++import json+from unittest.mock import patch++import httpx+import pytest++import litellm+from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler++_SSE_CHUNK: str = "data: " + json.dumps(+ {+ "candidates": [+ {+ "content": {"role": "model", "parts": [{"text": "hi"}]},+ "finishReason": "STOP",+ }+ ],+ "usageMetadata": {"promptTokenCount": 1, "candidatesTokenCount": 1, "totalTokenCount": 2},+ }+)++# Deliberately far from the per-request timeout so a fallback is detectable.+_CLIENT_DEFAULT_TIMEOUT: float = 600.0+_REQUEST_TIMEOUT: float = 2.0+++def _sse_response(request: httpx.Request) -> httpx.Response:+ return httpx.Response(+ 200,+ headers={"content-type": "text/event-stream"},+ content=(_SSE_CHUNK + "\n\n").encode(),+ request=request,+ )+++def _recording_sync_handler(seen_timeouts: list[dict]) -> HTTPHandler:+ def handler(request: httpx.Request) -> httpx.Response:+ seen_timeouts.append(request.extensions["timeout"])+ return _sse_response(request)++ return HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(handler), timeout=_CLIENT_DEFAULT_TIMEOUT))+++def _consume_sync_stream(**kwargs) -> str:+ response = litellm.completion(+ model="gemini/gemini-2.5-flash",+ messages=[{"role": "user", "content": "hello"}],+ api_key="fake-api-key",+ stream=True,+ timeout=_REQUEST_TIMEOUT,+ **kwargs,+ )+ return "".join(chunk.choices[0].delta.content or "" for chunk in response)+++def _assert_request_timeout(seen_timeouts: list[dict]) -> None:+ assert len(seen_timeouts) == 1+ assert seen_timeouts[0]["read"] == _REQUEST_TIMEOUT+ assert seen_timeouts[0]["connect"] == _REQUEST_TIMEOUT+++def test_sync_streaming_forwards_timeout_with_default_client():+ """The issue repro: no client passed, request goes through the module-level client."""+ seen_timeouts: list[dict] = []+ with patch.object(litellm, "module_level_client", _recording_sync_handler(seen_timeouts)):+ content = _consume_sync_stream()++ assert content == "hi"+ _assert_request_timeout(seen_timeouts)+++def test_sync_streaming_forwards_timeout_with_user_client():+ seen_timeouts: list[dict] = []+ content = _consume_sync_stream(client=_recording_sync_handler(seen_timeouts))++ assert content == "hi"+ _assert_request_timeout(seen_timeouts)+++def test_sync_streaming_stall_raises_with_configured_timeout_in_message():+ """A stalled stream fails with a Timeout whose message names the real timeout, not ``None``."""++ def handler(request: httpx.Request) -> httpx.Response:+ raise httpx.ReadTimeout("stalled", request=request)++ client = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(handler), timeout=_CLIENT_DEFAULT_TIMEOUT))+ with pytest.raises(litellm.Timeout) as exc_info:+ _consume_sync_stream(client=client, max_retries=0)++ assert "after None seconds" not in str(exc_info.value)+ assert f"after {_REQUEST_TIMEOUT} seconds" in str(exc_info.value)+++def test_sync_http_handler_timeout_message_reports_client_timeout_when_not_passed():+ def handler(request: httpx.Request) -> httpx.Response:+ raise httpx.ReadTimeout("stalled", request=request)++ client = HTTPHandler(client=httpx.Client(transport=httpx.MockTransport(handler), timeout=7.0))+ for method in (client.post, client.patch, client.put, client.delete):+ with pytest.raises(litellm.Timeout) as exc_info:+ method("https://example.com")+ assert "None" not in str(exc_info.value)+ assert "7.0" in str(exc_info.value)+++@pytest.mark.asyncio+async def test_async_streaming_forwards_timeout_with_user_client():+ seen_timeouts: list[dict] = []++ async def handler(request: httpx.Request) -> httpx.Response:+ seen_timeouts.append(request.extensions["timeout"])+ return _sse_response(request)++ client = AsyncHTTPHandler(timeout=_CLIENT_DEFAULT_TIMEOUT, transport=httpx.MockTransport(handler))+ response = await litellm.acompletion(+ model="gemini/gemini-2.5-flash",+ messages=[{"role": "user", "content": "hello"}],+ api_key="fake-api-key",+ stream=True,+ timeout=_REQUEST_TIMEOUT,+ client=client,+ )+ content = "".join([chunk.choices[0].delta.content or "" async for chunk in response])++ assert content == "hi"+ _assert_request_timeout(seen_timeouts)
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.pyindex 517ac9e98b1..109dddf44ed 100644@@ -2748,6 +2748,7 @@ async def make_call( model: str, messages: list, logging_obj,+ timeout: float | httpx.Timeout | None = None, ): if gemini_client is not None: client = gemini_client@@ -2757,7 +2758,9 @@ async def make_call( ) try:- response: Final = await client.post(api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj)+ response: Final = await client.post(+ api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj, timeout=timeout+ ) response.raise_for_status() except httpx.HTTPStatusError as e: exception_string: Final = str(await e.response.aread())@@ -2800,13 +2803,16 @@ def make_sync_call( model: str, messages: list, logging_obj,+ timeout: float | httpx.Timeout | None = None, ): if gemini_client is not None: client = gemini_client if client is None: client = HTTPHandler() # Create a new client if none provided - response: Final = client.post(api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj)+ response: Final = client.post(+ api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj, timeout=timeout+ ) if response.status_code != 200 and response.status_code != 201: raise VertexAIError(@@ -2924,6 +2930,7 @@ class VertexLLM(VertexBase): model=model, messages=messages, logging_obj=logging_obj,+ timeout=timeout, ), model=model, custom_llm_provider="vertex_ai_beta",@@ -3200,6 +3207,7 @@ class VertexLLM(VertexBase): messages=messages, logging_obj=logging_obj, headers=headers,+ timeout=timeout, ), model=model, custom_llm_provider="vertex_ai_beta",tests/integration/sdk/test_vertex_gemini_stream_timeout_wire.pynew file mode 100644index 00000000000..68d98961dce@@ -0,0 +1,147 @@+from __future__ import annotations++import asyncio+import json+import threading+import time+from collections.abc import Callable+from concurrent.futures import Future, ThreadPoolExecutor+from concurrent.futures import TimeoutError as FutureTimeout+from typing import Final++import pytest+from integration._support.vertex import service_account_json+from integration._support.wire import Reply, Request, Wire, wire_server++import litellm+from litellm.exceptions import MidStreamFallbackError++_MODEL: Final = "gemini-3.8-flash"+_PROJECT: Final = "scripted-project"+_LOCATION: Final = "us-east5"+_MESSAGES: Final = [{"role": "user", "content": "does the per-request timeout reach a stalled stream?"}]+_REQUEST_TIMEOUT_SECONDS: Final = 1.0+_HANG_DEADLINE_SECONDS: Final = 15.0+_TOKEN_REPLY: Final = Reply(+ body=json.dumps({"access_token": "scripted-token", "token_type": "Bearer", "expires_in": 3600}).encode()+)+++def _sse(text: str, finish_reason: str | None) -> bytes:+ candidate: Final = {+ "content": {"role": "model", "parts": [{"text": text}]},+ **({"finishReason": finish_reason} if finish_reason else {}),+ }+ usage: Final = {"promptTokenCount": 9, "candidatesTokenCount": 2, "totalTokenCount": 11}+ return b"data: " + json.dumps({"candidates": [candidate], "usageMetadata": usage}).encode() + b"\n\n"+++def _stream_that_stalls_after_the_first_chunk() -> Reply:+ return Reply(+ content_type="text/event-stream",+ chunks=(_sse("partial", None), _sse("", "STOP")),+ gate_after_first=threading.Event(),+ )+++def _stream_that_completes() -> Reply:+ return Reply(content_type="text/event-stream", chunks=(_sse("scripted ", None), _sse("answer", "STOP")))+++def _body_that_stalls_half_way() -> Reply:+ return Reply(chunks=(b'{"candidates": [', b"]}"), gate_after_first=threading.Event())+++def _peer(reply: Reply) -> Callable[[Request], Reply]:+ def respond(request: Request) -> Reply:+ if request.target == "/_oauth/token":+ return _TOKEN_REPLY+ assert request.method == "POST", request.method+ return reply++ return respond+++def _deployment(route: str, wire: Wire) -> dict[str, str]:+ if route == "gemini":+ return {"model": f"gemini/{_MODEL}", "api_base": wire.url, "api_key": "scripted-key"}+ return {+ "model": f"vertex_ai/{_MODEL}",+ "api_base": wire.url,+ "vertex_project": _PROJECT,+ "vertex_location": _LOCATION,+ "vertex_credentials": service_account_json(_PROJECT, wire.url),+ }+++def _text_of(chunk: object) -> str:+ choices: Final = getattr(chunk, "choices", None) or ()+ return "".join(str(choice.delta.content or "") for choice in choices)+++def _complete_sync(route: str, wire: Wire, stream: bool) -> str:+ response: Final = litellm.completion(+ **_deployment(route, wire), messages=_MESSAGES, stream=stream, timeout=_REQUEST_TIMEOUT_SECONDS+ )+ if stream:+ return "".join(_text_of(chunk) for chunk in response)+ return str(response.choices[0].message.content)+++async def _complete_async(route: str, wire: Wire, stream: bool) -> str:+ response: Final = await litellm.acompletion(+ **_deployment(route, wire), messages=_MESSAGES, stream=stream, timeout=_REQUEST_TIMEOUT_SECONDS+ )+ if stream:+ parts: Final = [_text_of(chunk) async for chunk in response]+ return "".join(parts)+ return str(response.choices[0].message.content)+++def _complete(client: str, route: str, wire: Wire, stream: bool) -> str:+ if client == "async":+ return asyncio.run(_complete_async(route, wire, stream))+ return _complete_sync(route, wire, stream)+++def _before_the_hang_deadline(call: Callable[[], str]) -> str:+ pool: Final = ThreadPoolExecutor(max_workers=1)+ future: Final[Future[str]] = pool.submit(call)+ pool.shutdown(wait=False)+ try:+ return future.result(timeout=_HANG_DEADLINE_SECONDS)+ except FutureTimeout:+ pytest.fail(+ f"still waiting on the peer {_HANG_DEADLINE_SECONDS}s after a {_REQUEST_TIMEOUT_SECONDS}s request timeout"+ )+++_CLIENTS: Final = ("sync", "async")+_ROUTES: Final = ("vertex_ai", "gemini")+++@pytest.mark.parametrize("route", _ROUTES)+@pytest.mark.parametrize("client", _CLIENTS)+def test_stalled_stream_raises_at_the_request_timeout(client: str, route: str) -> None:+ with wire_server(_peer(_stream_that_stalls_after_the_first_chunk())) as wire:+ started: Final = time.monotonic()+ with pytest.raises(MidStreamFallbackError):+ _before_the_hang_deadline(lambda: _complete(client, route, wire, stream=True))+ elapsed: Final = time.monotonic() - started+ assert elapsed >= _REQUEST_TIMEOUT_SECONDS, elapsed+ assert len(wire.drain()) >= 1+++@pytest.mark.parametrize("client", _CLIENTS)+def test_stalled_body_raises_at_the_request_timeout(client: str) -> None:+ with wire_server(_peer(_body_that_stalls_half_way())) as wire:+ started: Final = time.monotonic()+ with pytest.raises(litellm.Timeout):+ _before_the_hang_deadline(lambda: _complete(client, "vertex_ai", wire, stream=False))+ assert time.monotonic() - started >= _REQUEST_TIMEOUT_SECONDS+++@pytest.mark.parametrize("client", _CLIENTS)+def test_stream_with_a_request_timeout_still_completes(client: str) -> None:+ with wire_server(_peer(_stream_that_completes())) as wire:+ assert _complete(client, "vertex_ai", wire, stream=True) == "scripted answer"