Backtestslitellmrun-20261018-182052#41835
[Bug]: Vertex Grok (vertex_ai/xai/grok-*) is rejected on /v1beta/models/{model}:generateContent while /v1/chat/completions on the same deployment works
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the root cause the issue names: the generateContent config selector now consults the same route determination chat routing uses (get_vertex_ai_model_route), so vertex_ai/xai/grok-* returns None and falls through to the completion adapter. The independent tests confirm both make all four failing Grok scenarios pass with no regressions to Gemini, Claude, or chat completions. Patch B is a tight, two-file change that does exactly what the issue's 'narrow fix' paragraph asks, plus focused tests: an end-to-end mocked agenerate_content and agenerate_content_stream for vertex_ai/xai/grok-4.6 asserting the upstream URL is the openapi chat/completions endpoint and the Gemini-shaped output joins to '144', and a parametrized config-selection test. Its one gap is not forwarding litellm_params (base_model), so a numeric tuned-Gemini endpoint would now take the adapter path instead of native generateContent; that still works functionally but is a small behavior shift. Patch A's core fix is arguably more principled (uses_vertex_gemini_api, forwards base_model, covers gemma/agent_engine), but it is buried in a very large, mostly unrelated change set: a rewrite of cost_per_token prefix stripping in cost_calculator.py, a rewrite of the Databricks SSE iterator, changed exception semantics in the GenAI adapter handler (no longer wraps in ValueError, marks logging complete), a finish-chunk-holding mechanism in the stream wrapper, new stream_options injection in proxy request processing, a VertexAITokenCounter early-return, and ~1,500 lines of integration tests that depend on infrastructure outside the issue. Several of these (cost calculation, error types, token counting) alter behavior for callers unrelated to Grok and carry real regression risk, and they obscure the actual fix. Under the stated weighting, B resolves the issue at its root with targeted, non-tautological tests and minimal risk, while A's extras count against it rather than for it.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
16m 12s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
180vs 2,768 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
A proxy with a single Vertex Model Garden Grok deployment (vertex_ai/xai/grok-4.6) serves that model correctly on /v1/chat/completions, but the same model_name on the unified Gemini-style endpoint POST /v1beta/models/grok-4.6:generateContent fails.
On the same proxy and the same endpoint:
gemini-*deployments succeed.- Vertex Claude deployments succeed (they are translated to their own chat protocol).
vertex_ai/xai/grok-*deployments fail.
Expected: /v1beta/models/{model}:generateContent should serve a Vertex xai/grok-* deployment the same way it already serves other non-Gemini Vertex publisher models (Claude, Mistral, gpt-oss, Qwen, MiniMax, Moonshot, Z.ai, Gemma MaaS) — by translating the request to that model's chat-completions protocol and returning a normal candidates[...] response — instead of treating Grok as a native Gemini model.
Suspected cause (for maintainers): two different checks decide how a Vertex model is served, and they disagree about the xai/ publisher.
litellm/llms/vertex_ai/common_utils.py::get_vertex_ai_model_route()has an explicitmodel.startswith("xai/")branch returningMODEL_GARDEN, so chat completions work.litellm/utils.py::ProviderConfigManager.get_provider_google_genai_generate_content_config()falls back to thelitellm.completion()adapter only whenVertexAIPartnerModels.is_vertex_partner_model()is true, andPartnerModelPrefixeshas noxai/entry, so Grok is handed toVertexAIGoogleGenAIConfig(native Gemini) instead.
Note that simply adding xai/ to PartnerModelPrefixes would regress chat completions: is_vertex_partner_model() is evaluated before the xai/ Model Garden branch, and the partner handler raises Unknown partner model for xai/. The narrow fix is to make the generateContent handler selection use the same Model Garden determination that chat routing already uses.
User Flow
Before a (hypothetical) fix — the flow breaks the first time the user sends a Grok request to the Gemini-style URL, even though every other model at that same URL works and Grok itself works at the OpenAI-style URL.
- The user lists the gateway catalog with
GET http://localhost:4000/v1/modelsand sees bothgemini-2.5-flashandgrok-4.6offered. - The user sends
POST http://localhost:4000/v1/chat/completionswith{"model": "grok-4.6", "messages": [{"role": "user", "content": "What is 12*12? Reply with only the number."}]}and gets200withchoices[0].message.content=144. - The user points their Google GenAI SDK app at the gateway and sends
POST http://localhost:4000/v1beta/models/gemini-2.5-flash:generateContentwith{"contents": [...]}and gets200withcandidates[0].content.parts[0].text=144. - The user sends the identical body to
POST http://localhost:4000/v1beta/models/claude-opus-4-5:generateContentfor their Vertex Claude deployment and gets200with the samecandidates[...]shape, confirming the endpoint is not Gemini-only. - The user sends the identical body to
POST http://localhost:4000/v1beta/models/grok-4.6:generateContentand instead of200gets an error naming the Gemini publisher path for a model that is not a Gemini publisher model. - The user retries streaming at
POST http://localhost:4000/v1beta/models/grok-4.6:streamGenerateContentand gets the same failure, with no chunks. - The user has to keep a second client stack on the OpenAI-style route just for Grok, and cannot offer Grok to Google GenAI SDK / Gemini CLI users at all.
After a (hypothetical) fix — the same task completes entirely on the Gemini-style URL, for Grok just like for every other model in the catalog.
- The user lists the gateway catalog with
GET http://localhost:4000/v1/modelsand sees bothgemini-2.5-flashandgrok-4.6offered. - The user sends
POST http://localhost:4000/v1/chat/completionswith{"model": "grok-4.6", "messages": [{"role": "user", "content": "What is 12*12? Reply with only the number."}]}and gets200withchoices[0].message.content=144. - The user points their Google GenAI SDK app at the gateway and sends
POST http://localhost:4000/v1beta/models/gemini-2.5-flash:generateContentwith{"contents": [...]}and gets200withcandidates[0].content.parts[0].text=144. - The user sends the identical body to
POST http://localhost:4000/v1beta/models/claude-opus-4-5:generateContentfor their Vertex Claude deployment and gets200with the samecandidates[...]shape, confirming the endpoint is not Gemini-only. - The user sends the identical body to
POST http://localhost:4000/v1beta/models/grok-4.6:generateContentand gets200withcandidates[0].content.parts[0].text=144. - The user retries streaming at
POST http://localhost:4000/v1beta/models/grok-4.6:streamGenerateContentand receives streamed chunks that concatenate to144. - The user serves Grok and Gemini to the same Google GenAI SDK / Gemini CLI clients with one base URL and no per-model special casing.
Proof the bug occurs
Proxy setup
config.yaml (vanilla LiteLLM 1.102.0rc1, no custom hooks):
model_list:
- model_name: grok-4.6
litellm_params:
model: vertex_ai/xai/grok-4.6
vertex_project: os.environ/VERTEXAI_PROJECT
vertex_location: global
vertex_credentials: os.environ/VERTEXAI_JSON_FILEPATH
- model_name: gemini-2.5-flash
litellm_params:
model: vertex_ai/gemini-2.5-flash
vertex_project: os.environ/VERTEXAI_PROJECT
vertex_location: global
vertex_credentials: os.environ/VERTEXAI_JSON_FILEPATH
- model_name: claude-haiku-4.5
litellm_params:
model: vertex_ai/claude-haiku-4-5@20251001
vertex_project: os.environ/VERTEXAI_PROJECT
vertex_location: global
vertex_credentials: os.environ/VERTEXAI_JSON_FILEPATH
general_settings:
master_key: sk-...
Env vars the proxy ran with (secrets redacted):
export VERTEXAI_PROJECT=vertexai-eng
export VERTEXAI_JSON_FILEPATH=/path/to/sa.json # service-account JSON not pasted
export LITELLM_MASTER_KEY=sk-...
# version captured:
python -c 'from importlib.metadata import version; print(version("litellm"))'
# -> 1.102.0rc1
litellm --config config.yaml --port 4001 --detailed_debug
Step 2 — Grok on /v1/chat/completions (works)
curl -sS -i -X POST http://localhost:4001/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"model":"grok-4.6","temperature":0,"messages":[{"role":"user","content":"What is 12*12? Reply with only the number."}]}'
HTTP/1.1 200 OK
date: Fri, 18 Sep 2026 16:34:12 GMT
content-length: 669
content-type: application/json
x-litellm-call-id: 7c830719-6109-4ee7-8801-746d17ac35ec
x-litellm-model-id: 76e4f2b5962b73c15ed3790570a516ec437b63becec5d92804e9704c1024116f
x-litellm-model-name: vertex_ai/xai/grok-4.6
x-litellm-model-api-base: global-aiplatform.googleapis.com/v1/projects/vertexai-eng/locations/global/publishers/google/models/xai/grok-4.6:generateContent
x-litellm-version: 1.102.0rc1
x-litellm-response-cost: 0.000876
x-litellm-response-cost-original: 0.000876
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.00018199999999999998
x-litellm-response-cost-output: 0.00063
x-litellm-response-cost-cache-read: 6.4e-05
x-litellm-response-cost-reasoning: 0.00063
x-litellm-response-cost-tool-usage: 0.0
x-litellm-key-spend: 0.000876
x-litellm-response-duration-ms: 7779.472
x-litellm-callback-duration-ms: 0.0
x-litellm-model-group: grok-4.6
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 0
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff
{"id":"baca5a89-2306-4552-9d68-3e6c6d72d7b8","created":1789749262,"model":"grok-4.6","object":"chat.completion","system_fingerprint":"fp_grok46_0","choices":[{"finish_reason":"stop","index":0,"message":{"content":"144","role":"assistant"}}],"usage":{"completion_tokens":1,"prompt_tokens":219,"total_tokens":325,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":105,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":128,"text_tokens":219,"image_tokens":0},"cost_in_usd_ticks":0,"extra_properties":{"google":{"traffic_type":"ON_DEMAND"}},"num_sources_used":0},"service_tier":"default"}
Step 3 — Gemini on :generateContent (works)
curl -sS -i -X POST http://localhost:4001/v1beta/models/gemini-2.5-flash:generateContent \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"contents":[{"role":"user","parts":[{"text":"What is 12*12? Reply with only the number."}]}],"generationConfig":{"temperature":0}}'
HTTP/1.1 200 OK
date: Fri, 18 Sep 2026 16:34:22 GMT
content-length: 501
content-type: application/json
x-litellm-call-id: 6af107ab-5c24-4ec0-b6b4-d66985f391bb
x-litellm-model-id: 190e673dda51a2a74099cf445ec013bd387b6dfe873d38f78b299214d53a165f
x-litellm-model-name: vertex_ai/gemini-2.5-flash
x-litellm-model-api-base: global-aiplatform.googleapis.com/v1/projects/vertexai-eng/locations/global/publishers/google/models/gemini-2.5-flash:generateContent
x-litellm-version: 1.102.0rc1
x-litellm-response-cost: 0.000132
x-litellm-response-cost-original: 0.000132
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 4.5e-06
x-litellm-response-cost-output: 0.0001275
x-litellm-response-cost-reasoning: 0.00012000000000000002
x-litellm-response-cost-tool-usage: 0.0
x-litellm-key-spend: 0.000132
x-litellm-response-duration-ms: 1273.658
x-litellm-callback-duration-ms: 0.0
x-litellm-model-group: gemini-2.5-flash
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 0
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff
{"candidates":[{"content":{"role":"model","parts":[{"text":"144"}]},"finishReason":"STOP","avgLogprobs":-1.6457966168721516}],"usageMetadata":{"promptTokenCount":15,"candidatesTokenCount":3,"totalTokenCount":66,"trafficType":"ON_DEMAND","promptTokensDetails":[{"modality":"TEXT","tokenCount":15}],"candidatesTokensDetails":[{"modality":"TEXT","tokenCount":3}],"thoughtsTokenCount":48},"modelVersion":"gemini-2.5-flash","createTime":"2026-09-18T16:34:23.464810Z","responseId":"D2itaqqvHNeKirgP-ruU8A4"}
Step 4 — Vertex Claude on :generateContent (works)
curl -sS -i -X POST http://localhost:4001/v1beta/models/claude-haiku-4.5:generateContent \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"contents":[{"role":"user","parts":[{"text":"What is 12*12? Reply with only the number."}]}],"generationConfig":{"temperature":0}}'
HTTP/1.1 200 OK
date: Fri, 18 Sep 2026 16:34:23 GMT
content-length: 220
content-type: application/json
x-litellm-call-id: 89112682-8240-443f-b1c7-e6d84e68fefe
x-litellm-version: 1.102.0rc1
x-litellm-response-cost-original: 0.0
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 0.0
x-litellm-response-cost-tool-usage: 0.0
x-litellm-key-spend: 0.0
x-litellm-response-duration-ms: 708.321
x-litellm-model-group: claude-haiku-4.5
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 0
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff
{"candidates":[{"content":{"parts":[{"text":"144"}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[]}],"usageMetadata":{"promptTokenCount":20,"candidatesTokenCount":5,"totalTokenCount":25},"text":"144"}
Step 5 — Grok on :generateContent (fails)
curl -sS -i -X POST http://localhost:4001/v1beta/models/grok-4.6:generateContent \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"contents":[{"role":"user","parts":[{"text":"What is 12*12? Reply with only the number."}]}],"generationConfig":{"temperature":0}}'
HTTP/1.1 404 Not Found
date: Fri, 18 Sep 2026 16:34:24 GMT
x-litellm-call-id: d42cda37-bedc-4f3a-a192-5a65c25a619b
x-litellm-version: 1.102.0rc1
x-litellm-response-cost: 0
x-litellm-key-spend: 0.0
x-litellm-timeout: 600.0
content-length: 194
content-type: application/json
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff
{"error":{"message":"litellm.NotFoundError: Vertex_aiException - . Received Model Group=grok-4.6\nAvailable Model Group Fallbacks=None","type":"invalid_request_error","param":null,"code":"404"}}
Relevant --detailed_debug lines showing the upstream URL LiteLLM called for Grok:
16:34:25 - LiteLLM Router:INFO: router.py:5315 - ageneric_api_call_with_fallbacks(model=grok-4.6)[31m Exception litellm.NotFoundError: Vertex_aiException - [0m
16:34:25 - LiteLLM Router:INFO: router.py:7396 - Trying to fallback b/w models
16:34:25 - LiteLLM Proxy:ERROR: common_request_processing.py:1456 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.NotFoundError: Vertex_aiException - . Received Model Group=grok-4.6
Available Model Group Fallbacks=None
Traceback (most recent call last):
File "/app/venv/lib64/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 11752, in async_generate_content_handler
response = await async_httpx_client.post(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/litellm/litellm_core_utils/logging_utils.py", line 336, in async_wrapper
result: Final = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 807, in post
await _raise_masked_async_error(e, stream)
File "/app/venv/lib64/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 506, in _raise_masked_async_error
raise MaskedHTTPStatusError(e, message=_text, text=_text) from None
litellm.llms.custom_httpx.http_handler.MaskedHTTPStatusError: Client error '404 Not Found' for url 'https://aiplatform.googleapis.com/v1beta1/projects/vertexai-eng/locations/global/publishers/google/models/xai/grok-4.6:generateContent'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/venv/lib64/python3.12/site-packages/litellm/google_genai/main.py", line 287, in agenerate_content
response = await init_response
^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 11759, in async_generate_content_handler
raise self._handle_error(
^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 6076, in _handle_error
raise provider_error
litellm.llms.base_llm.chat.transformation.BaseLLMException
During handling of the above exception, ... (litellm_truncated skipped 2717 chars. Truncation is a stdout logging safeguard. Full, untruncated data is logged to logging callbacks (OTEL, Datadog, etc.) and at DEBUG level. To increase the truncation limit, set `MAX_STRING_LENGTH_STDOUT_LOG` in your env.) ...n should_retry_this_error
raise error
File "/app/venv/lib64/python3.12/site-packages/litellm/router.py", line 7705, in async_function_with_retries
response = await self.make_call(original_function, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/litellm/router.py", line 7875, in make_call
response = await response
^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/litellm/router.py", line 5322, in _ageneric_api_call_with_fallbacks_helper
raise e
File "/app/venv/lib64/python3.12/site-packages/litellm/router.py", line 5295, in _ageneric_api_call_with_fallbacks_helper
response = await response
^^^^^^^^^^^^^^
File "/app/venv/lib64/python3.12/site-packages/li
Smoking gun from that traceback:
Client error '404 Not Found' for url 'https://aiplatform.googleapis.com/v1beta1/projects/vertexai-eng/locations/global/publishers/google/models/xai/grok-4.6:generateContent'
Step 6 — Grok streaming on :streamGenerateContent (fails)
curl -sS -i -X POST "http://localhost:4001/v1beta/models/grok-4.6:streamGenerateContent?alt=sse" \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"contents":[{"role":"user","parts":[{"text":"What is 12*12? Reply with only the number."}]}],"generationConfig":{"temperature":0}}'
HTTP/1.1 404 Not Found
date: Fri, 18 Sep 2026 16:34:24 GMT
x-litellm-call-id: deba670d-8f44-4f8e-85e6-ce0164379620
x-litellm-version: 1.102.0rc1
x-litellm-response-cost: 0
x-litellm-key-spend: 0.0
x-litellm-timeout: 600.0
content-length: 194
content-type: application/json
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff
{"error":{"message":"litellm.NotFoundError: Vertex_aiException - . Received Model Group=grok-4.6\nAvailable Model Group Fallbacks=None","type":"invalid_request_error","param":null,"code":"404"}}
Upstream URL from the same debug log:
Client error '404 Not Found' for url 'https://aiplatform.googleapis.com/v1beta1/projects/vertexai-eng/locations/global/publishers/google/models/xai/grok-4.6:streamGenerateContent?alt=sse'
Same failure via the Google GenAI SDK
from google import genai
from google.genai.types import GenerateContentConfig, HttpOptions
client = genai.Client(
api_key="sk-...",
http_options=HttpOptions(base_url="http://localhost:4001"),
)
print(client.models.generate_content(
model="grok-4.6",
contents="What is 12*12? Reply with only the number.",
config=GenerateContentConfig(temperature=0.0),
).text)
ClientError: 404 None. {'error': {'message': 'litellm.NotFoundError: Vertex_aiException - . Received Model Group=grok-4.6\nAvailable Model Group Fallbacks=None', 'type': 'invalid_request_error', 'param': None, 'code': '404'}}
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.102.0rc1
Twitter / LinkedIn details
_No response_
The two patches
litellm/llms/vertex_ai/common_utils.pyindex 588416ff28..d279eb2506 100644@@ -227,7 +227,7 @@ class VertexAIModelRoute(str, Enum): VERTEX_AI_MODEL_ROUTES: Final = [f"{route.value}/" for route in VertexAIModelRoute] -def get_vertex_ai_model_route(model: str, litellm_params: dict | None = None) -> VertexAIModelRoute:+def get_vertex_ai_model_route(model: str, litellm_params: Mapping[str, object] | None = None) -> VertexAIModelRoute: """ Determine which handler to use for a Vertex AI model based on the model name. @@ -259,9 +259,9 @@ def get_vertex_ai_model_route(model: str, litellm_params: dict | None = None) -> ) # Check base_model in litellm_params for gemini override- if litellm_params and litellm_params.get("base_model") is not None:- if "gemini" in litellm_params["base_model"]:- return VertexAIModelRoute.GEMINI+ base_model: Final = litellm_params.get("base_model") if litellm_params else None+ if isinstance(base_model, str) and "gemini" in base_model:+ return VertexAIModelRoute.GEMINI # Check for agent_engine models (Reasoning Engines) if "agent_engine/" in model:litellm/utils.pyindex b74f97772f..a3eb8d717f 100644@@ -10240,19 +10240,24 @@ class ProviderConfigManager: return GoogleGenAIConfig() elif litellm.LlmProviders.VERTEX_AI == provider:+ from litellm.llms.vertex_ai.common_utils import (+ VertexAIModelRoute,+ get_vertex_ai_model_route,+ ) from litellm.llms.vertex_ai.google_genai.transformation import ( VertexAIGoogleGenAIConfig, )- from litellm.llms.vertex_ai.vertex_ai_partner_models.main import (- VertexAIPartnerModels,- ) #########################################################- # If Vertex Partner models like Anthropic, Mistral, etc. are used,+ # If Vertex Partner models like Anthropic, Mistral, etc. or Model Garden+ # OpenAI-compatible publisher models (e.g. xai/grok-*) are used, # return None as we want this to go through the litellm.completion() adapter # and not the Google Gen AI adapter #########################################################- if VertexAIPartnerModels.is_vertex_partner_model(model):+ if get_vertex_ai_model_route(model=model) in (+ VertexAIModelRoute.PARTNER_MODELS,+ VertexAIModelRoute.MODEL_GARDEN,+ ): return None #########################################################tests/unit/google_genai/test_google_genai_main.pyindex 238fff7dec..564e2ca95a 100644@@ -259,3 +259,160 @@ async def test_native_fields_forwarded_on_async_stream(): body = mock_post.call_args.kwargs["json"] assert body["safetySettings"] == safety_settings assert "safetySettings" not in body.get("generationConfig", {})+++_GROK_CONTENTS = [+ {"role": "user", "parts": [{"text": "What is 12*12? Reply with only the number."}]}+]+++def _patch_vertex_model_garden_auth():+ from unittest.mock import patch++ return patch(+ "litellm.llms.vertex_ai.vertex_model_garden.main.VertexAIModelGardenModels._ensure_access_token",+ return_value=("fake-token", "test-project"),+ )+++@pytest.mark.asyncio+async def test_vertex_xai_grok_generate_content_uses_chat_completions_route():+ """Vertex Model Garden Grok must be served on :generateContent by translating+ to its OpenAI-compatible chat protocol, not by calling the Gemini publisher path+ (.../publishers/google/models/xai/grok-*:generateContent), which 404s."""+ from unittest.mock import AsyncMock, MagicMock, patch++ from litellm.google_genai.main import agenerate_content+ from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler++ upstream = MagicMock()+ upstream.status_code = 200+ upstream.headers = {}+ upstream.json.return_value = {+ "id": "chatcmpl-grok",+ "object": "chat.completion",+ "created": 1789749262,+ "model": "xai/grok-4.6",+ "choices": [+ {+ "index": 0,+ "message": {"role": "assistant", "content": "144"},+ "finish_reason": "stop",+ }+ ],+ "usage": {"prompt_tokens": 20, "completion_tokens": 1, "total_tokens": 21},+ }++ with (+ _patch_vertex_model_garden_auth(),+ patch.object(+ AsyncHTTPHandler, "post", new_callable=AsyncMock, return_value=upstream+ ) as mock_post,+ ):+ response = await agenerate_content(+ model="vertex_ai/xai/grok-4.6",+ contents=_GROK_CONTENTS,+ vertex_project="test-project",+ vertex_location="global",+ )++ mock_post.assert_called_once()+ called_url = mock_post.call_args.kwargs.get("url") or mock_post.call_args.args[0]+ assert called_url == (+ "https://aiplatform.googleapis.com/v1/projects/test-project/locations/global"+ "/endpoints/openapi/chat/completions"+ ), called_url+ assert json.loads(mock_post.call_args.kwargs["data"])["model"] == "xai/grok-4.6"+ assert response["candidates"][0]["content"]["parts"] == [{"text": "144"}], response+++@pytest.mark.asyncio+async def test_vertex_xai_grok_stream_generate_content_uses_chat_completions_route():+ """:streamGenerateContent for Vertex Grok must stream through the chat+ completions protocol and yield Gemini-shaped chunks that join to the answer."""+ from unittest.mock import AsyncMock, MagicMock, patch++ from litellm.google_genai.main import agenerate_content_stream+ from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler++ def _sse(content, finish_reason=None):+ return "data: " + json.dumps(+ {+ "id": "chatcmpl-grok",+ "object": "chat.completion.chunk",+ "created": 1789749262,+ "model": "xai/grok-4.6",+ "choices": [+ {+ "index": 0,+ "delta": {"content": content} if content else {},+ "finish_reason": finish_reason,+ }+ ],+ }+ )++ async def _lines():+ for line in (_sse("1"), _sse("44"), _sse(None, "stop"), "data: [DONE]"):+ yield line++ upstream = MagicMock()+ upstream.status_code = 200+ upstream.headers = {}+ upstream.aiter_lines = MagicMock(return_value=_lines())++ with (+ _patch_vertex_model_garden_auth(),+ patch.object(+ AsyncHTTPHandler, "post", new_callable=AsyncMock, return_value=upstream+ ) as mock_post,+ ):+ stream = await agenerate_content_stream(+ model="vertex_ai/xai/grok-4.6",+ contents=_GROK_CONTENTS,+ vertex_project="test-project",+ vertex_location="global",+ )+ text = ""+ async for chunk in stream:+ payload = chunk.decode() if isinstance(chunk, bytes) else chunk+ if isinstance(payload, str):+ for line in payload.splitlines():+ if line.startswith("data: "):+ payload_dict = json.loads(line[len("data: "):])+ for part in payload_dict["candidates"][0]["content"]["parts"]:+ text += part.get("text", "")+ else:+ for part in payload["candidates"][0]["content"]["parts"]:+ text += part.get("text", "")++ called_url = mock_post.call_args.kwargs.get("url") or mock_post.call_args.args[0]+ assert called_url.endswith("/endpoints/openapi/chat/completions"), called_url+ assert ":streamGenerateContent" not in called_url+ assert text == "144"+++@pytest.mark.parametrize(+ "model,uses_native_gemini_config",+ [+ ("xai/grok-4.6", False),+ ("claude-haiku-4-5@20251001", False),+ ("openai/gpt-oss-120b-maas", False),+ ("gemini-2.5-flash", True),+ ],+)+def test_vertex_generate_content_handler_matches_chat_route(+ model, uses_native_gemini_config+):+ """Only Gemini publisher models get the native Vertex generateContent config;+ Model Garden (xai/grok-*) and partner models go through the completion adapter."""+ from litellm.llms.vertex_ai.google_genai.transformation import (+ VertexAIGoogleGenAIConfig,+ )+ from litellm.utils import ProviderConfigManager++ config = ProviderConfigManager.get_provider_google_genai_generate_content_config(+ model=model, provider=litellm.LlmProviders.VERTEX_AI+ )++ assert isinstance(config, VertexAIGoogleGenAIConfig) is uses_native_gemini_config, config
litellm/cost_calculator.pyindex 961f61d6fca..43ce71af7b8 100644@@ -556,12 +556,11 @@ def cost_per_token( assert custom_llm_provider is not None # caller-supplied or get_llm_provider - model_without_prefix = model model_parts: Final = model.split("/", 1)- if len(model_parts) > 1:- model_without_prefix = model_parts[1]- else:- model_without_prefix = model+ model_is_provider_relative: Final = (+ model_with_provider == f"{custom_llm_provider}/{model}" and model_with_provider in model_cost_ref+ )+ model_without_prefix: Final = model if len(model_parts) == 1 or model_is_provider_relative else model_parts[1] """ Code block that formats model to lookup in litellm.model_cost Option1. model = "bedrock/ap-northeast-1/anthropic.claude-instant-v1". This is the most accurate since it is region based. Should always be option 1litellm/google_genai/adapters/handler.pyindex ed06aca0809..7caa0c62571 100644@@ -1,7 +1,10 @@ from collections.abc import AsyncIterator, Coroutine, Mapping from typing import Final, cast +from openai.types.chat import ChatCompletionStreamOptionsParam+ import litellm+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj from litellm.types.google_genai.adapters import GenerateContentCompletionKwargs from litellm.types.router import GenericLiteLLMParams from litellm.types.utils import ModelResponse@@ -12,6 +15,23 @@ from .transformation import GoogleGenAIAdapter GOOGLE_GENAI_ADAPTER: Final = GoogleGenAIAdapter() +def _mark_failure_logged_by_inner_completion(adapter_kwargs: Mapping[str, object]) -> None:+ outer_logging_obj: Final = adapter_kwargs.get("litellm_logging_obj")+ if isinstance(outer_logging_obj, LiteLLMLoggingObj):+ outer_logging_obj.mark_logging_complete(event_type="sync_failure")+ outer_logging_obj.mark_logging_complete(event_type="async_failure")+++def _adapter_stream_options(requested: object) -> ChatCompletionStreamOptionsParam | None:+ match requested:+ case {"include_usage": True}:+ return {"include_usage": True}+ case Mapping():+ return None+ case _:+ return {"include_usage": True}++ class GenerateContentToCompletionHandler: """Handler for transforming generate_content calls to completion format when provider config is None""" @@ -47,9 +67,14 @@ class GenerateContentToCompletionHandler: completion_kwargs["extra_headers"] = extra_kwargs["extra_headers"] if "proxy_server_request" in extra_kwargs: completion_kwargs["proxy_server_request"] = extra_kwargs["proxy_server_request"]+ if "litellm_metadata" in extra_kwargs:+ completion_kwargs["litellm_metadata"] = extra_kwargs["litellm_metadata"] if stream: completion_kwargs["stream"] = stream+ stream_options: Final = _adapter_stream_options((extra_kwargs or {}).get("stream_options"))+ if stream_options is not None:+ completion_kwargs["stream_options"] = stream_options return GenerateContentCompletionKwargs(**completion_kwargs) @@ -75,33 +100,33 @@ class GenerateContentToCompletionHandler: try: completion_response: Final = await litellm.acompletion(**completion_kwargs)+ except Exception:+ _mark_failure_logged_by_inner_completion(kwargs)+ raise - if stream:- # Check if completion_response is actually a stream or a ModelResponse- # This can happen in error cases or when stream is not properly supported- if not hasattr(completion_response, "__aiter__"):- # If it's not a stream, treat it as a regular response- generate_content_response = GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content(- cast(ModelResponse, completion_response)- )- return generate_content_response- else:- # Transform streaming completion response to generate_content format- transformed_stream: Final = GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(- completion_response- )- if transformed_stream is not None:- return transformed_stream- raise ValueError("Failed to transform streaming response")- else:- # Transform completion response back to generate_content format+ if stream:+ # Check if completion_response is actually a stream or a ModelResponse+ # This can happen in error cases or when stream is not properly supported+ if not hasattr(completion_response, "__aiter__"):+ # If it's not a stream, treat it as a regular response generate_content_response = GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content( cast(ModelResponse, completion_response) ) return generate_content_response-- except Exception as e:- raise ValueError(f"Error calling litellm.acompletion for generate_content: {e}")+ else:+ # Transform streaming completion response to generate_content format+ transformed_stream: Final = GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(+ completion_response+ )+ if transformed_stream is not None:+ return transformed_stream+ raise ValueError("Failed to transform streaming response")+ else:+ # Transform completion response back to generate_content format+ generate_content_response = GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content(+ cast(ModelResponse, completion_response)+ )+ return generate_content_response @staticmethod def generate_content_handler(@@ -136,30 +161,30 @@ class GenerateContentToCompletionHandler: try: completion_response: Final = litellm.completion(**completion_kwargs)+ except Exception:+ _mark_failure_logged_by_inner_completion(kwargs)+ raise - if stream:- # Check if completion_response is actually a stream or a ModelResponse- # This can happen in error cases or when stream is not properly supported- if not hasattr(completion_response, "__iter__"):- # If it's not a stream, treat it as a regular response- generate_content_response = GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content(- cast(ModelResponse, completion_response)- )- return generate_content_response- else:- # Transform streaming completion response to generate_content format- transformed_stream: Final = GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(- completion_response- )- if transformed_stream is not None:- return transformed_stream- raise ValueError("Failed to transform streaming response")- else:- # Transform completion response back to generate_content format+ if stream:+ # Check if completion_response is actually a stream or a ModelResponse+ # This can happen in error cases or when stream is not properly supported+ if not hasattr(completion_response, "__iter__"):+ # If it's not a stream, treat it as a regular response generate_content_response = GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content( cast(ModelResponse, completion_response) ) return generate_content_response-- except Exception as e:- raise ValueError(f"Error calling litellm.completion for generate_content: {e}")+ else:+ # Transform streaming completion response to generate_content format+ transformed_stream: Final = GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(+ completion_response+ )+ if transformed_stream is not None:+ return transformed_stream+ raise ValueError("Failed to transform streaming response")+ else:+ # Transform completion response back to generate_content format+ generate_content_response = GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content(+ cast(ModelResponse, completion_response)+ )+ return generate_content_responselitellm/google_genai/adapters/transformation.pyindex 10340eb7acf..3333b44b216 100644@@ -158,8 +158,17 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper): self.sent_first_chunk = False self.accumulated_tool_calls = dict[int, _ToolCallAccumulator]() self._returned_response = False+ self.finish_chunk_awaiting_usage: Mapping[str, object] | None = None super().__init__(completion_stream) + def hold_finish_chunk_for_usage(self, finish_chunk: Mapping[str, object]) -> None:+ self.finish_chunk_awaiting_usage = finish_chunk++ def take_finish_chunk_awaiting_usage(self) -> Mapping[str, object] | None:+ finish_chunk: Final = self.finish_chunk_awaiting_usage+ self.finish_chunk_awaiting_usage = None+ return finish_chunk+ def __next__(self): try: if not hasattr(self.completion_stream, "__iter__"):@@ -176,6 +185,9 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper): if transformed_chunk: return transformed_chunk + finish_chunk: Final = self.take_finish_chunk_awaiting_usage()+ if finish_chunk:+ return finish_chunk raise StopIteration except StopIteration: raise@@ -198,6 +210,10 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper): if transformed_chunk: return transformed_chunk + finish_chunk: Final = self.take_finish_chunk_awaiting_usage()+ if finish_chunk:+ return finish_chunk+ # After the stream is exhausted, check for any remaining accumulated tool calls if self.accumulated_tool_calls: try:@@ -286,6 +302,10 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper): else: yield str(chunk).encode() + finish_chunk: Final = self.take_finish_chunk_awaiting_usage()+ if finish_chunk:+ yield f"data: {json.dumps(finish_chunk)}\n\n".encode()+ class GoogleGenAIAdapter: """Adapter for transforming Google GenAI generate_content requests to/from litellm.completion format"""@@ -657,8 +677,7 @@ class GoogleGenAIAdapter: # Extract the main response content from streaming chunk choice: Final = response.choices[0] if response.choices else None if not choice:- # Return empty chunk if no choices- return None+ return self._usage_only_streaming_chunk(response, wrapper) # Handle streaming choice if isinstance(choice, StreamingChoices):@@ -675,7 +694,7 @@ class GoogleGenAIAdapter: # Only create response chunk if we have parts or it's the final chunk if not parts and not finish_reason:- return None+ return self._usage_only_streaming_chunk(response, wrapper) # Create Google GenAI streaming format response streaming_chunk: Final[_JsonDict] = {@@ -689,20 +708,34 @@ class GoogleGenAIAdapter: ] } - # Add usage metadata only in the final chunk (when finish_reason is present)- if finish_reason:- usage_metadata: Final = (- self._map_usage(getattr(response, "usage", None))- if hasattr(response, "usage") and getattr(response, "usage", None)- else {- "promptTokenCount": 0,- "candidatesTokenCount": 0,- "totalTokenCount": 0,- }- )- streaming_chunk["usageMetadata"] = usage_metadata+ if not finish_reason:+ return streaming_chunk - return streaming_chunk+ finish_usage: Final[object] = getattr(response, "usage", None)+ if finish_usage:+ return {**streaming_chunk, "usageMetadata": self._map_usage(finish_usage)}++ wrapper.hold_finish_chunk_for_usage(+ {+ **streaming_chunk,+ "usageMetadata": {"promptTokenCount": 0, "candidatesTokenCount": 0, "totalTokenCount": 0},+ }+ )+ return None++ def _usage_only_streaming_chunk(+ self, response: ModelResponse | ModelResponseStream, wrapper: GoogleGenAIStreamWrapper+ ) -> Mapping[str, object] | None:+ usage: Final[object] = getattr(response, "usage", None)+ if not usage:+ return None+ finish_chunk: Final = wrapper.take_finish_chunk_awaiting_usage()+ if finish_chunk:+ return {**finish_chunk, "usageMetadata": self._map_usage(usage)}+ return {+ "candidates": [{"content": {"parts": [{"text": ""}], "role": "model"}, "index": 0}],+ "usageMetadata": self._map_usage(usage),+ } def _transform_openai_message_to_google_genai_parts( self,litellm/google_genai/main.pyindex 9ab3d78471c..f74d78aeac1 100644@@ -145,6 +145,7 @@ class GenerateContentHelper: ProviderConfigManager.get_provider_google_genai_generate_content_config( model=model, provider=litellm.LlmProviders(custom_llm_provider),+ litellm_params=litellm_params.model_dump(include={"base_model", "api_base"}), ) ) litellm/llms/databricks/streaming_utils.pyindex c729e6fcc00..eca87f9dd5f 100644@@ -11,6 +11,18 @@ from litellm.types.llms.openai import ( from litellm.types.utils import GenericStreamingChunk, Usage +def _usage_block(usage: Usage) -> ChatCompletionUsageBlock:+ return ChatCompletionUsageBlock(+ prompt_tokens=usage.prompt_tokens or 0,+ completion_tokens=usage.completion_tokens or 0,+ total_tokens=usage.total_tokens or 0,+ prompt_tokens_details=usage.prompt_tokens_details.model_dump() if usage.prompt_tokens_details else None,+ completion_tokens_details=(+ usage.completion_tokens_details.model_dump() if usage.completion_tokens_details else None+ ),+ )++ class ModelResponseIterator: def __init__(self, streaming_response, sync_stream: bool): self.streaming_response = streaming_response@@ -29,21 +41,13 @@ class ModelResponseIterator: # arrives with an empty ``choices`` list — return usage without # indexing ``choices[0]``. if len(processed_chunk.choices) == 0:- final_usage: Final = getattr(processed_chunk, "usage", None)+ final_usage: Final[Usage | None] = getattr(processed_chunk, "usage", None) return GenericStreamingChunk( text="", tool_use=None, is_finished=False, finish_reason="",- usage=(- ChatCompletionUsageBlock(- prompt_tokens=final_usage.prompt_tokens or 0,- completion_tokens=final_usage.completion_tokens or 0,- total_tokens=final_usage.total_tokens or 0,- )- if final_usage is not None- else None- ),+ usage=_usage_block(final_usage) if final_usage is not None else None, index=0, ) @@ -72,11 +76,7 @@ class ModelResponseIterator: usage_chunk: Final[Usage | None] = getattr(processed_chunk, "usage", None) if usage_chunk is not None:- usage = ChatCompletionUsageBlock(- prompt_tokens=usage_chunk.prompt_tokens,- completion_tokens=usage_chunk.completion_tokens,- total_tokens=usage_chunk.total_tokens,- )+ usage = _usage_block(usage_chunk) return GenericStreamingChunk( text=text,@@ -97,42 +97,24 @@ class ModelResponseIterator: def __next__(self): if not hasattr(self, "response_iterator"): self.response_iterator = self.streaming_response+ chunk: Final = self._next_sse_data() try:- chunk = self.response_iterator.__next__()- except StopIteration:- raise StopIteration- except ValueError as e:- raise RuntimeError(f"Error receiving chunk from stream: {e}")-- try:- chunk = litellm.CustomStreamWrapper.strip_sse_data_from_chunk(chunk) or ""- chunk = chunk.strip()- if len(chunk) > 0:- json_chunk: Final = json.loads(chunk)- return self.chunk_parser(chunk=json_chunk)- else:- return GenericStreamingChunk(- text="",- is_finished=False,- finish_reason="",- usage=None,- index=0,- tool_use=None,- )- except StopIteration:- raise StopIteration+ json_chunk: Final = json.loads(chunk)+ return self.chunk_parser(chunk=json_chunk) except ValueError as e: verbose_logger.debug( "Error parsing chunk: %s,\nReceived chunk: %s. Defaulting to empty chunk here.", e, chunk )- return GenericStreamingChunk(- text="",- is_finished=False,- finish_reason="",- usage=None,- index=0,- tool_use=None,- )+ return _empty_generic_chunk()++ def _next_sse_data(self) -> str:+ try:+ for raw_chunk in self.response_iterator:+ if sse_data := _sse_data(raw_chunk):+ return sse_data+ except ValueError as e:+ raise RuntimeError(f"Error receiving chunk from stream: {e}")+ raise StopIteration # Async iterator def __aiter__(self):@@ -140,43 +122,38 @@ class ModelResponseIterator: return self async def __anext__(self):- try:- chunk = await self.async_response_iterator.__anext__()- except StopAsyncIteration:+ chunk: Final = await self._anext_sse_data()+ if chunk == "[DONE]": raise StopAsyncIteration- except ValueError as e:- raise RuntimeError(f"Error receiving chunk from stream: {e}")- except Exception as e:- raise RuntimeError(f"Error receiving chunk from stream: {e}")- try:- chunk = litellm.CustomStreamWrapper.strip_sse_data_from_chunk(chunk) or ""- chunk = chunk.strip()- if chunk == "[DONE]":- raise StopAsyncIteration- if len(chunk) > 0:- json_chunk: Final = json.loads(chunk)- return self.chunk_parser(chunk=json_chunk)- else:- return GenericStreamingChunk(- text="",- is_finished=False,- finish_reason="",- usage=None,- index=0,- tool_use=None,- )- except StopAsyncIteration:- raise StopAsyncIteration+ json_chunk: Final = json.loads(chunk)+ return self.chunk_parser(chunk=json_chunk) except ValueError as e: verbose_logger.debug( "Error parsing chunk: %s,\nReceived chunk: %s. Defaulting to empty chunk here.", e, chunk )- return GenericStreamingChunk(- text="",- is_finished=False,- finish_reason="",- usage=None,- index=0,- tool_use=None,- )+ return _empty_generic_chunk()++ async def _anext_sse_data(self) -> str:+ try:+ async for raw_chunk in self.async_response_iterator:+ if sse_data := _sse_data(raw_chunk):+ return sse_data+ except Exception as e:+ raise RuntimeError(f"Error receiving chunk from stream: {e}")+ raise StopAsyncIteration+++def _sse_data(raw_chunk: str) -> str:+ return (litellm.CustomStreamWrapper.strip_sse_data_from_chunk(raw_chunk) or "").strip()+++def _empty_generic_chunk() -> GenericStreamingChunk:+ return GenericStreamingChunk(+ text="",+ is_finished=False,+ finish_reason="",+ usage=None,+ index=0,+ tool_use=None,+ )litellm/llms/vertex_ai/common_utils.pyindex 588416ff282..381986877c0 100644@@ -227,7 +227,7 @@ class VertexAIModelRoute(str, Enum): VERTEX_AI_MODEL_ROUTES: Final = [f"{route.value}/" for route in VertexAIModelRoute] -def get_vertex_ai_model_route(model: str, litellm_params: dict | None = None) -> VertexAIModelRoute:+def get_vertex_ai_model_route(model: str, litellm_params: Mapping[str, object] | None = None) -> VertexAIModelRoute: """ Determine which handler to use for a Vertex AI model based on the model name. @@ -258,10 +258,9 @@ def get_vertex_ai_model_route(model: str, litellm_params: dict | None = None) -> VertexAIPartnerModels, ) - # Check base_model in litellm_params for gemini override- if litellm_params and litellm_params.get("base_model") is not None:- if "gemini" in litellm_params["base_model"]:- return VertexAIModelRoute.GEMINI+ base_model: Final = litellm_params.get("base_model") if litellm_params else None+ if isinstance(base_model, str) and "gemini" in base_model:+ return VertexAIModelRoute.GEMINI # Check for agent_engine models (Reasoning Engines) if "agent_engine/" in model:@@ -299,6 +298,13 @@ def get_vertex_ai_model_route(model: str, litellm_params: dict | None = None) -> return VertexAIModelRoute.NON_GEMINI +GEMINI_API_ROUTES: Final = frozenset({VertexAIModelRoute.GEMINI, VertexAIModelRoute.NON_GEMINI})+++def uses_vertex_gemini_api(model: str, litellm_params: Mapping[str, object] | None = None) -> bool:+ return get_vertex_ai_model_route(model, litellm_params) in GEMINI_API_ROUTES++ def get_supports_system_message( model: str, custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"] ) -> bool:@@ -1285,6 +1291,11 @@ class VertexAITokenCounter(BaseTokenCounter): deployment = deployment or {} count_tokens_params_request: Final = copy.deepcopy(deployment.get("litellm_params", {})) + if not VertexAIPartnerModels.is_vertex_partner_model(model_to_use) and not uses_vertex_gemini_api(+ model_to_use, count_tokens_params_request+ ):+ return None+ # Check if this is a partner model (Claude, Mistral, etc.) if VertexAIPartnerModels.is_vertex_partner_model(model_to_use): # Use partner models token counterlitellm/proxy/common_request_processing.pyindex 58e8866ce99..e0d6538a545 100644@@ -544,6 +544,8 @@ def stream_usage_tracking_updates( if isinstance(stream_options, dict) and "include_usage" not in stream_options: return {**scrub, "stream_options": {**stream_options, "include_usage": True}} return scrub+ if always_include is False and route_type == "agenerate_content_stream" and "stream_options" not in data:+ return {**scrub, "stream_options": {"include_usage": False}} if always_include is False or route_type != "acompletion": return scrub if isinstance(stream_options, dict) and stream_options.get("include_usage") is True:litellm/types/google_genai/adapters.pyindex 771b362cae3..f03c67aa01a 100644@@ -1,4 +1,7 @@-from typing_extensions import TypedDict+from collections.abc import Mapping++from openai.types.chat import ChatCompletionStreamOptionsParam+from typing_extensions import ReadOnly, TypedDict from litellm.types.llms.openai import ( AllMessageValues,@@ -18,5 +21,7 @@ class GenerateContentCompletionKwargs(TypedDict, total=False): tool_choice: ChatCompletionToolChoiceStringValues stream: bool metadata: dict[str, object]+ litellm_metadata: ReadOnly[Mapping[str, object]]+ stream_options: ReadOnly[ChatCompletionStreamOptionsParam] extra_headers: dict[str, str] | None proxy_server_request: dict[str, object] | Nonelitellm/utils.pyindex b74f97772fa..22aa4baf52e 100644@@ -10232,6 +10232,7 @@ class ProviderConfigManager: def get_provider_google_genai_generate_content_config( model: str, provider: LlmProviders,+ litellm_params: Mapping[str, object] | None = None, ) -> BaseGoogleGenAIGenerateContentConfig | None: if litellm.LlmProviders.GEMINI == provider: from litellm.llms.gemini.google_genai.transformation import (@@ -10240,25 +10241,13 @@ class ProviderConfigManager: return GoogleGenAIConfig() elif litellm.LlmProviders.VERTEX_AI == provider:+ from litellm.llms.vertex_ai.common_utils import uses_vertex_gemini_api from litellm.llms.vertex_ai.google_genai.transformation import ( VertexAIGoogleGenAIConfig, )- from litellm.llms.vertex_ai.vertex_ai_partner_models.main import (- VertexAIPartnerModels,- ) - #########################################################- # If Vertex Partner models like Anthropic, Mistral, etc. are used,- # return None as we want this to go through the litellm.completion() adapter- # and not the Google Gen AI adapter- #########################################################- if VertexAIPartnerModels.is_vertex_partner_model(model):+ if not uses_vertex_gemini_api(model, litellm_params): return None-- #########################################################- # If the model is not a Vertex Partner model, return the Vertex AI Google Gen AI Config- # This is for Vertex `gemini` models- ######################################################### return VertexAIGoogleGenAIConfig() return None tests/integration/providers/test_google_genai_adapter_wire.pyindex 6cb5bb0349f..7dd3ba7b109 100644@@ -1,10 +1,12 @@ import json import uuid from collections.abc import Mapping+from hashlib import sha256 from typing import Final import pytest-from integration._support.client import Gateway+from integration._support.client import Gateway, eventually, held+from integration._support.database import read_rows from integration._support.wire import Reply, Request, wire_server from pydantic import JsonValue, TypeAdapter @@ -437,6 +439,7 @@ def test_stream_generate_content_adapts_openai_sse_to_gemini_frames(gateway: Gat "model": "gpt-4o-mini", "messages": _expected_openai_messages(marker), "stream": True,+ "stream_options": {"include_usage": True}, "tools": _EXPECTED_TOOLS, "tool_choice": "required", }@@ -526,9 +529,6 @@ def test_generate_content_adapter_accepts_snake_case_generation_config(gateway: @pytest.mark.parametrize("prefix", _PREFIXES) @pytest.mark.parametrize("backend", ["openai", "anthropic"]) def test_stream_generate_content_adapter_reports_upstream_usage(gateway: Gateway, prefix: str, backend: str) -> None:- pytest.skip(- "BUG: native Google stream served by an openai or anthropic deployment ends with usageMetadata 0/0/0 although the upstream sent usage 20/9/29"- ) frames: Final = _openai_stream_frames() if backend == "openai" else _anthropic_stream_frames() def respond(request: Request) -> Reply:@@ -778,3 +778,280 @@ def test_generate_content_adapter_keeps_camel_case_inline_data(gateway: Gateway) body: Final = JSON_OBJECT.validate_json(response.content) assert body["candidates"] == _EXPECTED_CANDIDATES, response.text assert [(r.method, r.target) for r in wire.drain()] == [("POST", "/v1/chat/completions")], response.text+++_TEXT_BASE: Final = {+ "id": "chatcmpl-wire3",+ "object": "chat.completion.chunk",+ "created": 1700000000,+ "model": "gpt-4o-mini",+}+_TEXT_USAGE: Final = {"prompt_tokens": 13, "completion_tokens": 4, "total_tokens": 17}+_ZERO_USAGE: Final = {"promptTokenCount": 0, "candidatesTokenCount": 0, "totalTokenCount": 0}+++def _text_frame(chunk: Mapping[str, JsonValue], response_id: str = "chatcmpl-wire3") -> bytes:+ return f"data: {json.dumps({**_TEXT_BASE, 'id': response_id, **chunk})}\n\n".encode()+++_DELTA_CHUNK: Final = {+ "choices": [{"index": 0, "delta": {"role": "assistant", "content": "hello"}, "finish_reason": None}]+}+_FINISH_CHUNK: Final = {"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}+_USAGE_CHUNK: Final = {"choices": [], "usage": _TEXT_USAGE}+_TEXT_DELTA: Final = _text_frame(_DELTA_CHUNK)+_TEXT_FINISH: Final = _text_frame(_FINISH_CHUNK)+_TEXT_USAGE_FRAME: Final = _text_frame(_USAGE_CHUNK)+_TEXT_DONE: Final = b"data: [DONE]\n\n"+++def _billed_text_frames(include_usage: bool) -> tuple[bytes, ...]:+ response_id: Final = f"chatcmpl-{uuid.uuid4().hex}"+ chunks: Final = (_DELTA_CHUNK, _FINISH_CHUNK, *((_USAGE_CHUNK,) if include_usage else ()))+ return (*(_text_frame(chunk, response_id) for chunk in chunks), _TEXT_DONE)+++def _text_stream_request(marker: str, **fields: JsonValue) -> dict[str, JsonValue]:+ return {"contents": [{"role": "user", "parts": [{"text": f"say hello {marker}"}]}], **fields}+++def _gemini_frames(response_text: str) -> list[dict]:+ parts: Final = [part for part in response_text.split("\n\n") if part.strip()]+ assert all(part.startswith("data: ") for part in parts), response_text+ return [json.loads(part.removeprefix("data: ")) for part in parts]+++def _stop_frames(frames: list[dict]) -> list[dict]:+ return [frame for frame in frames if any(c.get("finishReason") for c in frame.get("candidates", []))]+++def _openai_text_model(scenario, wire) -> str:+ return scenario.model(model="openai/gpt-4o-mini", api_base=f"{wire.url}/v1", api_key="synthetic-openai-key")+++def _stream_text(gateway: Gateway, model: str, key: str, body: Mapping[str, JsonValue]):+ return gateway.client.post(+ f"/v1beta/models/{model}:streamGenerateContent?alt=sse",+ json=dict(body),+ headers={"Authorization": f"Bearer {key}"},+ )+++@pytest.mark.parametrize(+ ("caller_options", "forwarded"),+ [+ pytest.param(None, {"include_usage": True}, id="caller_sends_none"),+ pytest.param({"include_usage": True}, {"include_usage": True}, id="caller_opts_in"),+ pytest.param({"include_usage": False}, None, id="caller_opts_out"),+ pytest.param({}, None, id="caller_sends_empty_object"),+ ],+)+def test_stream_generate_content_adapter_forwards_the_callers_stream_options(+ gateway: Gateway, caller_options: dict | None, forwarded: dict | None+) -> None:+ def respond(request: Request) -> Reply:+ body: Final = json.loads(request.body)+ asks: Final = body.get("stream_options", {}).get("include_usage") is True+ frames: Final = (_TEXT_DELTA, _TEXT_FINISH, *((_TEXT_USAGE_FRAME,) if asks else ()), _TEXT_DONE)+ return Reply(content_type="text/event-stream", chunks=frames)++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _openai_text_model(scenario, wire)+ key: Final = scenario.key(models=[model])+ options: Final = {} if caller_options is None else {"stream_options": caller_options}+ response: Final = _stream_text(gateway, model, key, _text_stream_request(uuid.uuid4().hex, **options))+ assert response.status_code == 200, response.text+ frames: Final = _gemini_frames(response.text)+ stops: Final = _stop_frames(frames)+ assert len(stops) == 1 and stops[0] is frames[-1], response.text+ expected_usage: Final = (+ {"promptTokenCount": 13, "candidatesTokenCount": 4, "totalTokenCount": 17} if forwarded else _ZERO_USAGE+ )+ assert frames[-1]["usageMetadata"] == expected_usage, response.text+ received: Final = wire.drain()+ assert len(received) == 1+ assert json.loads(received[0].body).get("stream_options") == forwarded, received[0].body+++_BILLED_SELECT: Final = 'SELECT prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE api_key=%s'+++def test_stream_generate_content_adapter_without_upstream_usage_reports_the_billed_tokens(gateway: Gateway) -> None:+ def respond(_request: Request) -> Reply:+ return Reply(content_type="text/event-stream", chunks=_billed_text_frames(include_usage=False))++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _openai_text_model(scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_text(gateway, model, key, _text_stream_request(uuid.uuid4().hex))+ assert response.status_code == 200, response.text+ frames: Final = _gemini_frames(response.text)+ assert frames[0]["candidates"][0]["content"]["parts"] == [{"text": "hello"}], response.text+ stops: Final = _stop_frames(frames)+ assert len(stops) == 1 and stops[0] is frames[-1], response.text+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_BILLED_SELECT, (digest,)), lambda rows: len(rows) >= 1, seconds=70)+ rows: Final = held(lambda: read_rows(_BILLED_SELECT, (digest,)), lambda found: len(found) == 1, holding=3)+ usage: Final = frames[-1]["usageMetadata"]+ assert (usage["promptTokenCount"], usage["candidatesTokenCount"]) == (+ rows[0]["prompt_tokens"],+ rows[0]["completion_tokens"],+ ), (response.text, rows)+ assert usage["promptTokenCount"] > 0, response.text+ assert len(wire.drain()) == 1+++def test_stream_generate_content_adapter_usage_before_finish_keeps_the_real_usage_last(gateway: Gateway) -> None:+ def respond(_request: Request) -> Reply:+ return Reply(+ content_type="text/event-stream", chunks=(_TEXT_DELTA, _TEXT_USAGE_FRAME, _TEXT_FINISH, _TEXT_DONE)+ )++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _openai_text_model(scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_text(gateway, model, key, _text_stream_request(uuid.uuid4().hex))+ assert response.status_code == 200, response.text+ frames: Final = _gemini_frames(response.text)+ assert len(_stop_frames(frames)) == 1, response.text+ assert [frame["usageMetadata"] for frame in frames if "usageMetadata" in frame][-1] == {+ "promptTokenCount": 13,+ "candidatesTokenCount": 4,+ "totalTokenCount": 17,+ }, response.text+ assert len(wire.drain()) == 1+++def test_stream_generate_content_adapter_slow_usage_after_finish_still_arrives(gateway: Gateway) -> None:+ def respond(_request: Request) -> Reply:+ return Reply(+ content_type="text/event-stream",+ chunks=(_TEXT_DELTA, _TEXT_FINISH, _TEXT_USAGE_FRAME, _TEXT_DONE),+ pause_between_chunks=1.5,+ )++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _openai_text_model(scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_text(gateway, model, key, _text_stream_request(uuid.uuid4().hex))+ assert response.status_code == 200, response.text+ frames: Final = _gemini_frames(response.text)+ stops: Final = _stop_frames(frames)+ assert len(stops) == 1 and stops[0] is frames[-1], response.text+ assert frames[-1]["usageMetadata"] == {+ "promptTokenCount": 13,+ "candidatesTokenCount": 4,+ "totalTokenCount": 17,+ }, response.text+ assert len(wire.drain()) == 1+++def test_stream_generate_content_adapter_upstream_drop_after_finish_still_sends_stop(gateway: Gateway) -> None:+ def respond(_request: Request) -> Reply:+ return Reply(+ content_type="text/event-stream",+ chunks=(_TEXT_DELTA, _TEXT_FINISH, _TEXT_USAGE_FRAME, _TEXT_DONE),+ abort_after=2,+ pause_between_chunks=1.5,+ )++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _openai_text_model(scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_text(gateway, model, key, _text_stream_request(uuid.uuid4().hex))+ assert response.status_code == 200, response.text+ frames: Final = _gemini_frames(response.text)+ assert frames[0]["candidates"][0]["content"]["parts"] == [{"text": "hello"}], response.text+ assert len(_stop_frames(frames)) == 1, response.text+ assert len(wire.drain()) == 1+ healthy: Final = gateway.request("GET", "/health/liveliness")+ assert healthy.status_code == 200, healthy.text+++_ATTRIBUTION_SELECT: Final = (+ "SELECT request_tags, team_id, metadata->>'user_api_key_alias' AS alias, model_group, prompt_tokens,"+ ' completion_tokens FROM "LiteLLM_SpendLogs" WHERE api_key=%s'+)+_STATUS_SELECT: Final = 'SELECT status FROM "LiteLLM_SpendLogs" WHERE api_key=%s'+++@pytest.mark.parametrize("streamed", [False, True], ids=["generate", "stream"])+@pytest.mark.parametrize("status", [400, 401, 429])+def test_generate_content_adapter_upstream_error_keeps_the_provider_status(+ gateway: Gateway, status: int, streamed: bool+) -> None:+ message: Final = f"scripted openai rejection {uuid.uuid4().hex}"++ def respond(_request: Request) -> Reply:+ error: Final = {"message": message, "type": "invalid_request_error", "code": str(status)}+ return Reply(status=status, body=json.dumps({"error": error}).encode())++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = scenario.model(+ model="openai/gpt-4o-mini",+ api_base=f"{wire.url}/v1",+ api_key="synthetic-openai-key",+ num_retries=0,+ max_retries=0,+ )+ key: Final = scenario.key(models=[model])+ method: Final = "streamGenerateContent?alt=sse" if streamed else "generateContent"+ response: Final = gateway.client.post(+ f"/v1beta/models/{model}:{method}",+ json=_text_stream_request(uuid.uuid4().hex),+ headers={"Authorization": f"Bearer {key}"},+ )+ assert response.status_code == status, response.text+ assert message in response.text, response.text+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_STATUS_SELECT, (digest,)), lambda rows: len(rows) >= 1, seconds=70)+ rows: Final = held(lambda: read_rows(_STATUS_SELECT, (digest,)), lambda found: len(found) == 1, holding=3)+ assert rows[0]["status"] == "failure", rows+ assert len(wire.drain()) == 1+ healthy: Final = gateway.request("GET", "/health/liveliness")+ assert healthy.status_code == 200, healthy.text+++@pytest.mark.parametrize("streamed", [False, True], ids=["generate", "stream"])+def test_generate_content_adapter_spend_row_keeps_key_team_and_tag_attribution(+ gateway: Gateway, streamed: bool+) -> None:+ def respond(request: Request) -> Reply:+ if json.loads(request.body).get("stream") is True:+ return Reply(content_type="text/event-stream", chunks=_billed_text_frames(include_usage=True))+ return Reply(+ body=json.dumps(+ {+ "id": f"chatcmpl-{uuid.uuid4().hex}",+ "object": "chat.completion",+ "created": 1700000000,+ "model": "gpt-4o-mini",+ "choices": [+ {"index": 0, "message": {"role": "assistant", "content": "hello"}, "finish_reason": "stop"}+ ],+ "usage": _TEXT_USAGE,+ }+ ).encode()+ )++ marker: Final = uuid.uuid4().hex+ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _openai_text_model(scenario, wire)+ team: Final = scenario.team(models=[model])+ alias: Final = f"adapter-alias-{marker}"+ key: Final = scenario.key(models=[model], team_id=team, key_alias=alias)+ method: Final = "streamGenerateContent?alt=sse" if streamed else "generateContent"+ response: Final = gateway.client.post(+ f"/v1beta/models/{model}:{method}",+ json=_text_stream_request(marker),+ headers={"Authorization": f"Bearer {key}", "x-litellm-tags": f"adapter-tag-{marker}"},+ )+ assert response.status_code == 200, response.text+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_ATTRIBUTION_SELECT, (digest,)), lambda rows: len(rows) >= 1, seconds=70)+ rows: Final = held(lambda: read_rows(_ATTRIBUTION_SELECT, (digest,)), lambda found: len(found) == 1, holding=3)+ row: Final = rows[0]+ assert (row["team_id"], row["alias"], row["model_group"]) == (team, alias, model), row+ assert f"adapter-tag-{marker}" in json.dumps(row["request_tags"]), row+ assert (row["prompt_tokens"], row["completion_tokens"]) == (13, 4), row+ assert len(wire.drain()) == 1tests/integration/providers/test_google_genai_vertex_routes_wire.pynew file mode 100644index 00000000000..d2c2f3bafb2@@ -0,0 +1,499 @@+import json+import uuid+from collections.abc import Callable, Mapping, Sequence+from dataclasses import dataclass+from hashlib import sha256+from typing import Final++import httpx+import pytest+from integration._support.client import Gateway, Scenario, eventually, held, object_value+from integration._support.database import read_rows+from integration._support.vertex import service_account_json+from integration._support.wire import Reply, Request, Wire, wire_server+from pydantic import JsonValue, TypeAdapter++_JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])+_PROJECT: Final = "scripted-project"+_REPLY_TEXT: Final = "scripted vertex reply"+_PROMPT_TOKENS: Final = 11+_COMPLETION_TOKENS: Final = 7+_USAGE: Final = {+ "promptTokenCount": _PROMPT_TOKENS,+ "candidatesTokenCount": _COMPLETION_TOKENS,+ "totalTokenCount": _PROMPT_TOKENS + _COMPLETION_TOKENS,+}+_NATIVE_COUNT: Final = 31+_SPEND_SELECT: Final = (+ "SELECT request_id, call_type, model_group, status, cache_hit, prompt_tokens, completion_tokens, spend"+ ' FROM "LiteLLM_SpendLogs" WHERE api_key=%s ORDER BY "startTime"'+)+++@dataclass(frozen=True, slots=True)+class Route:+ model: str+ location: str+ adapter: bool+ endpoint_path: str = ""+ overrides: Mapping[str, JsonValue] | None = None+++_ROUTES: Final = {+ "model_garden_publisher": Route("xai/grok-4.20-non-reasoning", "global", adapter=True),+ "model_garden_endpoint": Route("openai/1234567890123456789", "us-central1", adapter=True),+ "gemma": Route(+ "gemma/gemma-2-2b-it",+ "us-central1",+ adapter=True,+ endpoint_path=f"/v1/projects/{_PROJECT}/locations/us-central1/endpoints/4242:predict",+ ),+ "gemini": Route("gemini-2.5-flash", "us-central1", adapter=False),+ "legacy_non_gemini": Route("chat-bison", "us-central1", adapter=False),+ "tuned_gemini_endpoint": Route(+ "9876543210", "us-central1", adapter=False, overrides={"base_model": "vertex_ai/gemini-2.5-flash"}+ ),+}+_ADAPTER_ROUTES: Final = tuple(name for name, route in _ROUTES.items() if route.adapter)+++def _openai_reply(model: str) -> dict[str, JsonValue]:+ return {+ "id": f"chatcmpl-{uuid.uuid4().hex}",+ "object": "chat.completion",+ "created": 1700000000,+ "model": model,+ "choices": [{"index": 0, "message": {"role": "assistant", "content": _REPLY_TEXT}, "finish_reason": "stop"}],+ "usage": {+ "prompt_tokens": _PROMPT_TOKENS,+ "completion_tokens": _COMPLETION_TOKENS,+ "total_tokens": _PROMPT_TOKENS + _COMPLETION_TOKENS,+ },+ }+++def _openai_frames(model: str, include_usage: bool) -> tuple[bytes, ...]:+ base: Final = {+ "id": f"chatcmpl-{uuid.uuid4().hex}",+ "object": "chat.completion.chunk",+ "created": 1700000000,+ "model": model,+ }++ def frame(chunk: Mapping[str, JsonValue]) -> bytes:+ return f"data: {json.dumps({**base, **chunk})}\n\n".encode()++ usage: Final = (frame({"choices": [], "usage": _openai_reply(model)["usage"]}),) if include_usage else ()+ return (+ frame(+ {"choices": [{"index": 0, "delta": {"role": "assistant", "content": _REPLY_TEXT}, "finish_reason": None}]}+ ),+ frame({"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}),+ *usage,+ b"data: [DONE]\n\n",+ )+++def _asks_for_usage(body: Mapping[str, JsonValue]) -> bool:+ options: Final = body.get("stream_options")+ return isinstance(options, dict) and options.get("include_usage") is True+++def _native_reply() -> dict[str, JsonValue]:+ return {+ "candidates": [+ {"content": {"parts": [{"text": _REPLY_TEXT}], "role": "model"}, "finishReason": "STOP", "index": 0}+ ],+ "usageMetadata": _USAGE,+ }+++def _native_frames() -> tuple[bytes, ...]:+ return (+ b'data: {"candidates":[{"content":{"parts":[{"text":"'+ + _REPLY_TEXT.encode()+ + b'"}],"role":"model"},"index":0}]}\n\n',+ b'data: {"candidates":[{"content":{"parts":[],"role":"model"},"finishReason":"STOP","index":0}],"usageMetadata":'+ + json.dumps(_USAGE).encode()+ + b"}\n\n",+ )+++def _publisher_not_found(request: Request) -> Reply:+ return Reply(+ status=404,+ body=json.dumps(+ {+ "error": {+ "code": 404,+ "message": f"Publisher Model {request.target} was not found",+ "status": "NOT_FOUND",+ }+ }+ ).encode(),+ )+++def _google_native_target(target: str) -> bool:+ return any(+ name in target for name in ("/models/gemini-2.5-flash:", "/models/chat-bison:", "/endpoints/9876543210:")+ )+++def _vertex_peer(status: int = 200, error: str = "scripted vertex rejection") -> Callable[[Request], Reply]:+ def respond(request: Request) -> Reply:+ if status != 200:+ return Reply(status=status, body=json.dumps({"error": {"message": error, "code": status}}).encode())+ if request.target.endswith("/chat/completions"):+ body: Final = _JSON_OBJECT.validate_json(request.body)+ model: Final = str(body.get("model", ""))+ if body.get("stream") is True:+ return Reply(content_type="text/event-stream", chunks=_openai_frames(model, _asks_for_usage(body)))+ return Reply(body=json.dumps(_openai_reply(model)).encode())+ if request.target.endswith(":predict"):+ return Reply(body=json.dumps({"predictions": _openai_reply("gemma-2-2b-it")}).encode())+ if not _google_native_target(request.target):+ return _publisher_not_found(request)+ if ":streamGenerateContent" in request.target:+ return Reply(content_type="text/event-stream", chunks=_native_frames())+ if request.target.endswith(":countTokens"):+ return Reply(body=json.dumps({"totalTokens": _NATIVE_COUNT}).encode())+ return Reply(body=json.dumps(_native_reply()).encode())++ return respond+++def _deployment(gateway: Gateway, scenario: Scenario, wire: Wire, route: Route, **extra: JsonValue) -> str:+ return scenario.model(+ model=f"vertex_ai/{route.model}",+ api_base=f"{wire.url}{route.endpoint_path}",+ api_key=None,+ vertex_project=_PROJECT,+ vertex_location=route.location,+ vertex_credentials=service_account_json(_PROJECT, gateway.upstream_url.rstrip("/")),+ **dict(route.overrides or {}),+ **extra,+ )+++def _contents(marker: str) -> list[JsonValue]:+ return [{"role": "user", "parts": [{"text": f"route probe {marker}"}]}]+++def _generate(gateway: Gateway, model: str, key: str, body: Mapping[str, JsonValue]) -> httpx.Response:+ return gateway.request("POST", f"/v1beta/models/{model}:generateContent", body, key=key)+++def _stream(gateway: Gateway, model: str, key: str, body: Mapping[str, JsonValue]) -> httpx.Response:+ return gateway.client.post(+ f"/v1beta/models/{model}:streamGenerateContent?alt=sse",+ json=dict(body),+ headers={"Authorization": f"Bearer {key}"},+ )+++def _frames(response: httpx.Response) -> list[dict[str, JsonValue]]:+ parts: Final = [part for part in response.text.split("\n\n") if part.strip()]+ assert all(part.startswith("data: ") for part in parts), response.text+ return [_JSON_OBJECT.validate_json(part.removeprefix("data: ")) for part in parts]+++def _finish_reasons(frames: Sequence[Mapping[str, JsonValue]]) -> list[JsonValue]:+ return [+ object_value(candidate).get("finishReason")+ for frame in frames+ for candidate in (frame.get("candidates") or []) # comprehension-ok: two-level frame walk+ if isinstance(candidate, dict) and candidate.get("finishReason")+ ]+++def _text(frames: Sequence[Mapping[str, JsonValue]]) -> str:+ return "".join(+ str(object_value(part).get("text", ""))+ for frame in frames+ for candidate in (frame.get("candidates") or []) # comprehension-ok: three-level frame walk+ for part in object_value(object_value(candidate).get("content") or {}).get("parts") or []+ )+++def _spend_rows(key: str, count: int) -> list[dict[str, JsonValue]]:+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_SPEND_SELECT, (digest,)), lambda values: len(values) >= count, seconds=70)+ return held(lambda: read_rows(_SPEND_SELECT, (digest,)), lambda values: len(values) == count, holding=3)+++def _route_targets(wire: Wire) -> list[tuple[str, str]]:+ return [(request.method, request.target) for request in wire.drain()]+++def _expected_adapter_target(route: Route) -> str:+ return route.endpoint_path if route.endpoint_path else "/chat/completions"+++def _cost_map_prices(gateway: Gateway, model: str) -> tuple[float, float]:+ entries: Final = gateway.get("/model/info")["data"]+ assert isinstance(entries, list)+ info: Final = next(+ object_value(object_value(entry)["model_info"])+ for entry in entries+ if object_value(entry)["model_name"] == model+ )+ prompt: Final = info.get("input_cost_per_token")+ completion: Final = info.get("output_cost_per_token")+ assert isinstance(prompt, float) and isinstance(completion, float) and prompt > 0 and completion > 0, info+ return prompt, completion+++@pytest.mark.parametrize("route_name", tuple(_ROUTES))+def test_generate_content_reaches_the_route_handler_chat_uses(gateway: Gateway, route_name: str) -> None:+ route: Final = _ROUTES[route_name]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ response: Final = _generate(gateway, model, key, {"contents": _contents(uuid.uuid4().hex)})+ assert response.status_code == 200, response.text+ body: Final = _JSON_OBJECT.validate_json(response.content)+ assert _text([body]) == _REPLY_TEXT, response.text+ assert _finish_reasons([body]) == ["STOP"], response.text+ assert body["usageMetadata"] == _USAGE, response.text+ targets: Final = _route_targets(wire)+ assert len(targets) == 1, targets+ if route.adapter:+ assert targets == [("POST", _expected_adapter_target(route))], targets+ else:+ assert targets[0][1].endswith(":generateContent"), targets+ rows: Final = _spend_rows(key, 1)+ assert rows[0]["model_group"] == model, rows+ assert (rows[0]["prompt_tokens"], rows[0]["completion_tokens"]) == (_PROMPT_TOKENS, _COMPLETION_TOKENS), rows+++@pytest.mark.parametrize("route_name", tuple(_ROUTES))+def test_stream_generate_content_ends_with_one_stop_carrying_upstream_usage(gateway: Gateway, route_name: str) -> None:+ route: Final = _ROUTES[route_name]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ response: Final = _stream(gateway, model, key, {"contents": _contents(uuid.uuid4().hex)})+ assert response.status_code == 200, response.text+ frames: Final = _frames(response)+ assert _text(frames) == _REPLY_TEXT, response.text+ assert _finish_reasons(frames) == ["STOP"], response.text+ assert _finish_reasons(frames[-1:]) == ["STOP"], response.text+ assert frames[-1]["usageMetadata"] == _USAGE, response.text+ received: Final = wire.drain()+ assert len(received) == 1, [request.target for request in received]+ if route.adapter and not route.endpoint_path:+ assert received[0].target == "/chat/completions", received[0].target+ assert _JSON_OBJECT.validate_json(received[0].body)["stream_options"] == {"include_usage": True}+ if not route.adapter:+ assert ":streamGenerateContent" in received[0].target, received[0].target+ assert "stream_options" not in _JSON_OBJECT.validate_json(received[0].body), received[0].body+ rows: Final = _spend_rows(key, 1)+ assert rows[0]["model_group"] == model, rows+ assert (rows[0]["prompt_tokens"], rows[0]["completion_tokens"]) == (_PROMPT_TOKENS, _COMPLETION_TOKENS), rows+++@pytest.mark.parametrize("route_name", _ADAPTER_ROUTES)+def test_count_tokens_counts_locally_on_adapter_routes(gateway: Gateway, route_name: str) -> None:+ route: Final = _ROUTES[route_name]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ response: Final = gateway.request(+ "POST", f"/v1beta/models/{model}:countTokens", {"contents": _contents("count")}, key=key+ )+ assert response.status_code == 200, response.text+ total: Final = _JSON_OBJECT.validate_json(response.content)["totalTokens"]+ assert isinstance(total, int) and 0 < total != _NATIVE_COUNT, response.text+ assert _route_targets(wire) == []+++@pytest.mark.parametrize("stream", [False, True], ids=["generate", "stream"])+def test_model_garden_spend_is_priced_from_the_cost_map(gateway: Gateway, stream: bool) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ prompt_price, completion_price = _cost_map_prices(gateway, model)+ call: Final = _stream if stream else _generate+ response: Final = call(gateway, model, key, {"contents": _contents(uuid.uuid4().hex)})+ assert response.status_code == 200, response.text+ expected: Final = _PROMPT_TOKENS * prompt_price + _COMPLETION_TOKENS * completion_price+ rows: Final = _spend_rows(key, 1)+ assert rows[0]["call_type"] == "acompletion", rows+ assert float(str(rows[0]["spend"])) == pytest.approx(expected), rows+ assert len(wire.drain()) == 1+++def test_model_garden_chat_and_generate_content_bill_the_same(gateway: Gateway) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ chat_key: Final = scenario.key(models=[model])+ genai_key: Final = scenario.key(models=[model])+ chat: Final = gateway.client.post(+ "/v1/chat/completions",+ json={"model": model, "messages": [{"role": "user", "content": "route probe"}], "stream": True},+ headers={"Authorization": f"Bearer {chat_key}"},+ )+ assert chat.status_code == 200, chat.text+ genai: Final = _stream(gateway, model, genai_key, {"contents": _contents("route probe")})+ assert genai.status_code == 200, genai.text+ chat_rows: Final = _spend_rows(chat_key, 1)+ genai_rows: Final = _spend_rows(genai_key, 1)+ assert (chat_rows[0]["prompt_tokens"], chat_rows[0]["completion_tokens"]) == (+ _PROMPT_TOKENS,+ _COMPLETION_TOKENS,+ ), chat_rows+ assert float(str(genai_rows[0]["spend"])) == pytest.approx(float(str(chat_rows[0]["spend"]))), (+ chat_rows,+ genai_rows,+ )+ assert float(str(genai_rows[0]["spend"])) > 0, genai_rows+ assert len(wire.drain()) == 2+++def _replayed_frames(response: httpx.Response, streamed: bool) -> list[dict[str, JsonValue]]:+ return _frames(response) if streamed else [_JSON_OBJECT.validate_json(response.content)]+++@pytest.mark.parametrize("streamed", [False, True], ids=["generate", "stream"])+@pytest.mark.parametrize("route_name", _ADAPTER_ROUTES)+def test_adapter_route_cache_hit_skips_the_upstream_and_keeps_attribution(+ gateway: Gateway, route_name: str, streamed: bool+) -> None:+ route: Final = _ROUTES[route_name]+ call: Final = _stream if streamed else _generate+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ body: Final = {"contents": _contents(uuid.uuid4().hex)}+ first: Final = call(gateway, model, key, body)+ assert first.status_code == 200, first.text+ second: Final = call(gateway, model, key, body)+ assert second.status_code == 200, second.text+ replayed: Final = _replayed_frames(second, streamed)+ assert _text(replayed) == _REPLY_TEXT, second.text+ assert replayed[-1]["usageMetadata"] == _replayed_frames(first, streamed)[-1]["usageMetadata"], second.text+ assert len(wire.drain()) == 1+ rows: Final = _spend_rows(key, 2)+ assert [row["cache_hit"] == "True" for row in rows] == [False, True], rows+ assert [row["model_group"] for row in rows] == [model, model], rows+ assert len({row["request_id"] for row in rows}) == 2, rows+++@pytest.mark.parametrize("stream", [False, True], ids=["generate", "stream"])+@pytest.mark.parametrize("status", [400, 401, 429])+def test_adapter_route_upstream_error_reaches_the_caller_with_its_status(+ gateway: Gateway, status: int, stream: bool+) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ message: Final = f"scripted vertex rejection {uuid.uuid4().hex}"+ with wire_server(_vertex_peer(status=status, error=message)) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route, num_retries=0)+ key: Final = scenario.key(models=[model])+ call: Final = _stream if stream else _generate+ response: Final = call(gateway, model, key, {"contents": _contents(uuid.uuid4().hex)})+ assert response.status_code == status, response.text+ assert message in response.text, response.text+ rows: Final = _spend_rows(key, 1)+ assert rows[0]["status"] == "failure", rows+ assert len(wire.drain()) >= 1+ healthy: Final = gateway.request("GET", "/health/liveliness")+ assert healthy.status_code == 200, healthy.text+++def test_adapter_route_failure_leaves_other_keys_serving(gateway: Gateway) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ with (+ wire_server(_vertex_peer(status=400)) as failing,+ wire_server(_vertex_peer()) as serving,+ gateway.scenario() as scenario,+ ):+ broken: Final = _deployment(gateway, scenario, failing, route, num_retries=0)+ working: Final = _deployment(gateway, scenario, serving, route)+ broken_key: Final = scenario.key(models=[broken])+ working_key: Final = scenario.key(models=[working])+ assert _generate(gateway, broken, broken_key, {"contents": _contents("a")}).status_code == 400+ served: Final = _generate(gateway, working, working_key, {"contents": _contents("b")})+ assert served.status_code == 200, served.text+ assert _spend_rows(working_key, 1)[0]["status"] == "success"+++@pytest.mark.parametrize(+ ("base_model", "adapter"),+ [+ pytest.param("vertex_ai/gemini-2.5-flash", False, id="gemini_alias_forces_native"),+ pytest.param("", True, id="empty_string"),+ pytest.param(None, True, id="null"),+ pytest.param("vertex_ai/xai/grok-4.20-non-reasoning", True, id="non_gemini_alias"),+ ],+)+def test_base_model_steers_generate_content_like_chat(gateway: Gateway, base_model: str | None, adapter: bool) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route, base_model=base_model)+ key: Final = scenario.key(models=[model])+ genai: Final = _generate(gateway, model, key, {"contents": _contents("alias")})+ genai_targets: Final = _route_targets(wire)+ chat: Final = gateway.request(+ "POST",+ "/v1/chat/completions",+ {"model": model, "messages": [{"role": "user", "content": "alias"}]},+ key=key,+ )+ chat_targets: Final = _route_targets(wire)+ assert len(genai_targets) == 1 and len(chat_targets) == 1, (genai_targets, chat_targets)+ assert (genai_targets[0][1] == "/chat/completions") is adapter, genai_targets+ assert (chat_targets[0][1] == "/chat/completions") is adapter, chat_targets+ assert genai.status_code == chat.status_code, (genai.text, chat.text)+++@pytest.mark.parametrize(+ "stream_options",+ [+ pytest.param(7, id="int"),+ pytest.param(["include_usage"], id="list"),+ pytest.param("", id="empty_string"),+ pytest.param("x" * 5000, id="five_kb"),+ ],+)+def test_malformed_stream_options_fall_back_to_usage(gateway: Gateway, stream_options: JsonValue) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ response: Final = _stream(+ gateway, model, key, {"contents": _contents(uuid.uuid4().hex), "stream_options": stream_options}+ )+ assert response.status_code == 200, response.text+ frames: Final = _frames(response)+ assert _finish_reasons(frames) == ["STOP"], response.text+ assert frames[-1]["usageMetadata"] == _USAGE, response.text+ received: Final = wire.drain()+ assert len(received) == 1+ assert _JSON_OBJECT.validate_json(received[0].body)["stream_options"] == {"include_usage": True}+++def test_duplicate_stream_options_key_keeps_the_last_value(gateway: Gateway) -> None:+ route: Final = _ROUTES["model_garden_publisher"]+ with wire_server(_vertex_peer()) as wire, gateway.scenario() as scenario:+ model: Final = _deployment(gateway, scenario, wire, route)+ key: Final = scenario.key(models=[model])+ contents: Final = json.dumps(_contents(uuid.uuid4().hex))+ raw: Final = (+ '{"contents": ' + contents + ', "stream_options": {"include_usage": true},'+ ' "stream_options": {"include_usage": false}}'+ )+ response: Final = gateway.client.post(+ f"/v1beta/models/{model}:streamGenerateContent?alt=sse",+ content=raw.encode(),+ headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"},+ )+ assert response.status_code == 200, response.text+ frames: Final = _frames(response)+ assert _finish_reasons(frames) == ["STOP"], response.text+ assert frames[-1]["usageMetadata"] == {"promptTokenCount": 0, "candidatesTokenCount": 0, "totalTokenCount": 0}+ received: Final = wire.drain()+ assert len(received) == 1+ assert "stream_options" not in _JSON_OBJECT.validate_json(received[0].body), received[0].bodytests/integration/sdk/test_google_genai_adapter_stream_usage_sdk.pynew file mode 100644index 00000000000..1f2c3c81938@@ -0,0 +1,253 @@+import asyncio+import json+import threading+from collections.abc import Callable, Iterator, Mapping, Sequence+from typing import Final++import pytest+from integration._support.client import eventually, held+from integration._support.vertex import service_account_json+from integration._support.wire import Reply, Request, Wire, wire_server+from pydantic import JsonValue, TypeAdapter++import litellm+from litellm.google_genai import agenerate_content, agenerate_content_stream, generate_content+from litellm.integrations.custom_logger import CustomLogger++_JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])+_PROVIDER_KEY: Final = "synthetic-provider-key"+_REPLY_TEXT: Final = "sdk reply"+_PROMPT_TOKENS: Final = 13+_COMPLETION_TOKENS: Final = 4+_CACHED_TOKENS: Final = 5+_USAGE: Final = {+ "prompt_tokens": _PROMPT_TOKENS,+ "completion_tokens": _COMPLETION_TOKENS,+ "total_tokens": _PROMPT_TOKENS + _COMPLETION_TOKENS,+ "prompt_tokens_details": {"cached_tokens": _CACHED_TOKENS},+}+_GEMINI_USAGE: Final = {+ "promptTokenCount": _PROMPT_TOKENS,+ "candidatesTokenCount": _COMPLETION_TOKENS,+ "totalTokenCount": _PROMPT_TOKENS + _COMPLETION_TOKENS,+}+_REJECTION: Final = "scripted key rejected"+_PROJECT: Final = "scripted-project"+_TOKEN: Final = json.dumps({"access_token": "scripted-token", "token_type": "Bearer", "expires_in": 3600}).encode()+++def _frame(chunk: Mapping[str, JsonValue]) -> bytes:+ body: Final = {+ "id": "chatcmpl-sdk",+ "object": "chat.completion.chunk",+ "created": 1700000000,+ "model": "m",+ **chunk,+ }+ return f"data: {json.dumps(body)}\n\n".encode()+++def _frames(include_usage: bool) -> tuple[bytes, ...]:+ usage: Final = (_frame({"choices": [], "usage": _USAGE}),) if include_usage else ()+ return (+ _frame({"choices": [{"index": 0, "delta": {"role": "assistant", "content": _REPLY_TEXT}}]}),+ b"\n\n",+ _frame({"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}),+ b"\n\n\n\n",+ *usage,+ b"data: [DONE]\n\n",+ )+++def _asks_for_usage(body: Mapping[str, JsonValue]) -> bool:+ options: Final = body.get("stream_options")+ return isinstance(options, dict) and options.get("include_usage") is True+++def _respond(status: int) -> Callable[[Request], Reply]:+ def respond(request: Request) -> Reply:+ if request.target.endswith("/_oauth/token"):+ return Reply(body=_TOKEN)+ if status != 200:+ return Reply(status=status, body=json.dumps({"error": {"message": _REJECTION, "code": status}}).encode())+ body: Final = _JSON_OBJECT.validate_json(request.body)+ return Reply(+ content_type="text/event-stream", chunks=_frames(_asks_for_usage(body) or "/ml/v1/" in request.target)+ )++ return respond+++@pytest.fixture+def wire() -> Iterator[Wire]:+ with wire_server(_respond(200)) as served:+ yield served+++class _FailureCounter(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self._lock: Final = threading.Lock()+ self._failures: Final[list[str]] = []++ def log_failure_event(+ self, kwargs: dict[str, object], response_obj: object, start_time: object, end_time: object+ ) -> None:+ with self._lock:+ self._failures.append("sync")++ async def async_log_failure_event(+ self, kwargs: dict[str, object], response_obj: object, start_time: object, end_time: object+ ) -> None:+ with self._lock:+ self._failures.append("async")++ def failures(self) -> tuple[str, ...]:+ with self._lock:+ return tuple(self._failures)+++def _logged_once(counter: _FailureCounter, kind: str) -> tuple[str, ...]:+ eventually(counter.failures, lambda seen: kind in seen, seconds=10)+ return held(counter.failures, lambda seen: seen.count(kind) == 1, holding=2)+++def _contents() -> list[JsonValue]:+ return [{"role": "user", "parts": [{"text": "sdk adapter probe"}]}]+++def _stop_usage(chunks: Sequence[Mapping[str, JsonValue]]) -> list[JsonValue]:+ return [+ chunk.get("usageMetadata")+ for chunk in chunks+ if any(isinstance(c, dict) and c.get("finishReason") for c in chunk.get("candidates") or [])+ ]+++def _model_requests(wire: Wire) -> list[Request]:+ return [request for request in wire.drain() if not request.target.endswith("/_oauth/token")]+++def _upstream_bodies(wire: Wire) -> list[dict[str, JsonValue]]:+ return [_JSON_OBJECT.validate_json(request.body) for request in _model_requests(wire)]+++def _member(name: str, wire: Wire) -> dict[str, JsonValue]:+ match name:+ case "vertex_model_garden":+ return {+ "model": "vertex_ai/xai/grok-4.20-non-reasoning",+ "api_base": wire.url,+ "vertex_project": _PROJECT,+ "vertex_location": "global",+ "vertex_credentials": service_account_json(_PROJECT, wire.url),+ }+ case "watsonx":+ return {+ "model": "watsonx/ibm/granite-3-8b-instruct",+ "api_base": wire.url,+ "zen_api_key": "synthetic-zen-key",+ "project_id": _PROJECT,+ }+ case "maritalk":+ return {"model": "maritalk/sabia-3", "api_base": wire.url, "api_key": _PROVIDER_KEY}+ case _:+ return {"model": "amazon_nova/nova-pro-v1", "api_base": wire.url, "api_key": _PROVIDER_KEY}+++def _chat_usage(chunks: Sequence[litellm.ModelResponseStream]) -> list[litellm.Usage]:+ return [usage for chunk in chunks if isinstance(usage := getattr(chunk, "usage", None), litellm.Usage)]+++@pytest.mark.parametrize("member", ["vertex_model_garden", "watsonx", "maritalk", "amazon_nova"])+def test_sync_completion_stream_keeps_usage_sent_after_finish(wire: Wire, member: str) -> None:+ stream: Final = litellm.completion(+ messages=[{"role": "user", "content": "sync iterator"}],+ stream=True,+ stream_options={"include_usage": True},+ num_retries=0,+ **_member(member, wire),+ )+ chunks: Final = tuple(stream)+ assert "".join(chunk.choices[0].delta.content or "" for chunk in chunks if chunk.choices) == _REPLY_TEXT+ usage: Final = _chat_usage(chunks)+ assert [(u.prompt_tokens, u.completion_tokens) for u in usage] == [(_PROMPT_TOKENS, _COMPLETION_TOKENS)], chunks+ assert len(_model_requests(wire)) == 1+++@pytest.mark.parametrize("member", ["vertex_model_garden", "watsonx"])+async def test_async_completion_stream_keeps_usage_sent_after_finish(wire: Wire, member: str) -> None:+ stream: Final = await litellm.acompletion(+ messages=[{"role": "user", "content": "async iterator"}],+ stream=True,+ stream_options={"include_usage": True},+ num_retries=0,+ **_member(member, wire),+ )+ chunks: Final = tuple([chunk async for chunk in stream])+ assert "".join(chunk.choices[0].delta.content or "" for chunk in chunks if chunk.choices) == _REPLY_TEXT+ usage: Final = _chat_usage(chunks)+ assert [(u.prompt_tokens, u.completion_tokens) for u in usage] == [(_PROMPT_TOKENS, _COMPLETION_TOKENS)], chunks+ assert len(_model_requests(wire)) == 1+++def _sse_chunks(frames: Sequence[bytes]) -> list[dict[str, JsonValue]]:+ parts: Final = [part for part in b"".join(frames).decode().split("\n\n") if part.strip()]+ assert all(part.startswith("data: ") for part in parts), parts+ return [_JSON_OBJECT.validate_json(part.removeprefix("data: ")) for part in parts]+++@pytest.mark.parametrize(+ ("stream_options", "asks"),+ [+ pytest.param(None, True, id="caller_sends_none"),+ pytest.param({"include_usage": True}, True, id="caller_opts_in"),+ pytest.param({"include_usage": False}, False, id="caller_opts_out"),+ ],+)+async def test_async_generate_content_stream_adapter_usage_follows_the_caller(+ wire: Wire, stream_options: JsonValue, asks: bool+) -> None:+ options: Final = {} if stream_options is None else {"stream_options": stream_options}+ stream: Final = await agenerate_content_stream(+ model="openai/gpt-5.4-mini", contents=_contents(), api_base=wire.url, api_key=_PROVIDER_KEY, **options+ )+ frames: Final = tuple([frame async for frame in stream])+ chunks: Final = _sse_chunks(frames)+ zero: Final = {"promptTokenCount": 0, "candidatesTokenCount": 0, "totalTokenCount": 0}+ assert _stop_usage(chunks) == [_GEMINI_USAGE if asks else zero], chunks+ bodies: Final = _upstream_bodies(wire)+ assert len(bodies) == 1 and bodies[0].get("stream_options") == ({"include_usage": True} if asks else None), bodies+++def test_sync_generate_content_adapter_error_keeps_the_provider_status_and_logs_once(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ counter: Final = _FailureCounter()+ monkeypatch.setattr(litellm, "callbacks", [counter])+ with wire_server(_respond(401)) as rejecting:+ with pytest.raises(litellm.AuthenticationError) as raised:+ generate_content(+ model="openai/gpt-5.4-mini", contents=_contents(), api_base=rejecting.url, api_key=_PROVIDER_KEY+ )+ assert raised.value.status_code == 401+ assert _REJECTION in str(raised.value)+ assert len(rejecting.drain()) == 1+ assert _logged_once(counter, "sync").count("sync") == 1+++async def test_async_generate_content_adapter_error_keeps_the_provider_status_and_logs_once(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ counter: Final = _FailureCounter()+ monkeypatch.setattr(litellm, "callbacks", [counter])+ with wire_server(_respond(401)) as rejecting:+ with pytest.raises(litellm.AuthenticationError) as raised:+ await agenerate_content(+ model="openai/gpt-5.4-mini", contents=_contents(), api_base=rejecting.url, api_key=_PROVIDER_KEY+ )+ assert raised.value.status_code == 401+ assert _REJECTION in str(raised.value)+ assert len(rejecting.drain()) == 1+ logged: Final = await asyncio.to_thread(_logged_once, counter, "async")+ assert logged.count("async") == 1, loggedtests/integration/streaming/test_google_genai_stream_usage_settings_wire.pynew file mode 100644index 00000000000..85017667c74@@ -0,0 +1,343 @@+import json+import re+import threading+import uuid+from collections.abc import Callable, Mapping, Sequence+from concurrent.futures import ThreadPoolExecutor+from contextlib import ExitStack+from hashlib import sha256+from pathlib import Path+from typing import Final++import httpx+import psutil+import pytest+import yaml+from integration._support.client import Gateway, eventually, held, object_value+from integration._support.database import read_rows+from integration._support.process import graceful_stop_seconds, owned_proxy_process+from integration._support.vertex import service_account_json+from integration._support.wire import Reply, Request, Wire, wire_server+from pydantic import JsonValue, TypeAdapter++_JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])+_PROJECT: Final = "scripted-project"+_ADAPTER_MODEL: Final = "settings-adapter-openai"+_NATIVE_MODEL: Final = "settings-native-gemini"+_REPLY_TEXT: Final = "settings reply"+_PROMPT_TOKENS: Final = 13+_COMPLETION_TOKENS: Final = 4+_OPENAI_USAGE: Final = {+ "prompt_tokens": _PROMPT_TOKENS,+ "completion_tokens": _COMPLETION_TOKENS,+ "total_tokens": _PROMPT_TOKENS + _COMPLETION_TOKENS,+}+_GEMINI_USAGE: Final = {+ "promptTokenCount": _PROMPT_TOKENS,+ "candidatesTokenCount": _COMPLETION_TOKENS,+ "totalTokenCount": _PROMPT_TOKENS + _COMPLETION_TOKENS,+}+_ZERO_USAGE: Final = {"promptTokenCount": 0, "candidatesTokenCount": 0, "totalTokenCount": 0}+_STARTED_WORKER: Final = re.compile(r"Started server process \[(\d+)\]")+_BURST_SELECT: Final = (+ 'SELECT api_key, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs"'+ " WHERE api_key = ANY(string_to_array(%s, ','))"+)+_CACHE_SELECT: Final = (+ 'SELECT cache_hit, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE api_key=%s ORDER BY "startTime"'+)+_CALLER_OPTIONS: Final = {+ "caller_sends_none": None,+ "caller_opts_in": {"include_usage": True},+ "caller_opts_out": {"include_usage": False},+ "caller_sends_empty_object": {},+}+_UPSTREAM_ASKS_FOR_USAGE: Final = {+ "always_off": {+ "caller_sends_none": False,+ "caller_opts_in": True,+ "caller_opts_out": False,+ "caller_sends_empty_object": False,+ },+ "always_on": {+ "caller_sends_none": True,+ "caller_opts_in": True,+ "caller_opts_out": False,+ "caller_sends_empty_object": True,+ },+}+_SETTING_VALUES: Final = {"always_off": False, "always_on": True}+++def _openai_frame(chunk: Mapping[str, JsonValue], response_id: str) -> bytes:+ body: Final = {"id": response_id, "object": "chat.completion.chunk", "created": 1700000000, **chunk}+ return f"data: {json.dumps(body)}\n\n".encode()+++def _openai_frames(include_usage: bool) -> tuple[bytes, ...]:+ response_id: Final = f"chatcmpl-{uuid.uuid4().hex}"+ usage: Final = (_openai_frame({"choices": [], "usage": _OPENAI_USAGE}, response_id),) if include_usage else ()+ return (+ _openai_frame({"choices": [{"index": 0, "delta": {"role": "assistant", "content": _REPLY_TEXT}}]}, response_id),+ _openai_frame({"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}, response_id),+ *usage,+ b"data: [DONE]\n\n",+ )+++def _native_frames() -> tuple[bytes, ...]:+ text: Final = {"candidates": [{"content": {"parts": [{"text": _REPLY_TEXT}], "role": "model"}, "index": 0}]}+ stop: Final = {+ "candidates": [{"content": {"parts": [], "role": "model"}, "finishReason": "STOP", "index": 0}],+ "usageMetadata": _GEMINI_USAGE,+ }+ return (f"data: {json.dumps(text)}\n\n".encode(), f"data: {json.dumps(stop)}\n\n".encode())+++def _asks_for_usage(body: Mapping[str, JsonValue]) -> bool:+ options: Final = body.get("stream_options")+ return isinstance(options, dict) and options.get("include_usage") is True+++def _peer(gate: threading.Event | None = None) -> Callable[[Request], Reply]:+ def respond(request: Request) -> Reply:+ if ":streamGenerateContent" in request.target:+ return Reply(content_type="text/event-stream", chunks=_native_frames())+ body: Final = _JSON_OBJECT.validate_json(request.body)+ return Reply(+ content_type="text/event-stream",+ chunks=_openai_frames(_asks_for_usage(body)),+ gate_after_first=gate,+ )++ return respond+++def _owned_config(path: Path, gateway: Gateway, wire: Wire, general: Mapping[str, JsonValue]) -> Path:+ config: Final = yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text())+ path.write_text(+ yaml.safe_dump(+ {+ **config,+ "model_list": [+ {+ "model_name": _ADAPTER_MODEL,+ "litellm_params": {+ "model": "openai/gpt-5.4-mini",+ "api_base": wire.url,+ "api_key": "synthetic-provider-key",+ },+ },+ {+ "model_name": _NATIVE_MODEL,+ "litellm_params": {+ "model": "vertex_ai/gemini-2.5-flash",+ "api_base": wire.url,+ "vertex_project": _PROJECT,+ "vertex_location": "us-central1",+ "vertex_credentials": service_account_json(_PROJECT, gateway.upstream_url.rstrip("/")),+ },+ },+ ],+ "general_settings": {**(config.get("general_settings") or {}), **general},+ }+ )+ )+ return path+++def _stream_body(marker: str, stream_options: JsonValue) -> dict[str, JsonValue]:+ base: Final = {"contents": [{"role": "user", "parts": [{"text": f"settings probe {marker}"}]}]}+ return base if stream_options is None else {**base, "stream_options": stream_options}+++def _stream(client: httpx.Client, model: str, key: str, body: Mapping[str, JsonValue]) -> httpx.Response:+ return client.post(+ f"/v1beta/models/{model}:streamGenerateContent?alt=sse",+ json=dict(body),+ headers={"Authorization": f"Bearer {key}"},+ )+++def _frames(response: httpx.Response) -> list[dict[str, JsonValue]]:+ parts: Final = [part for part in response.text.split("\n\n") if part.strip()]+ assert all(part.startswith("data: ") for part in parts), response.text+ return [_JSON_OBJECT.validate_json(part.removeprefix("data: ")) for part in parts]+++def _stop_frames(frames: Sequence[Mapping[str, JsonValue]]) -> list[Mapping[str, JsonValue]]:+ return [+ frame+ for frame in frames+ if any(object_value(candidate).get("finishReason") for candidate in frame.get("candidates") or [])+ ]+++def _model_bodies(wire: Wire) -> list[dict[str, JsonValue]]:+ return [_JSON_OBJECT.validate_json(request.body) for request in wire.drain()]+++def _assert_one_stop_with(response: httpx.Response, usage: Mapping[str, int]) -> None:+ assert response.status_code == 200, response.text+ stops: Final = _stop_frames(_frames(response))+ assert len(stops) == 1, response.text+ assert stops[0]["usageMetadata"] == usage, response.text+++def _assert_caller_option_reaches_upstream(+ client: httpx.Client, key: str, wire: Wire, setting: str, caller: str+) -> None:+ response: Final = _stream(client, _ADAPTER_MODEL, key, _stream_body(uuid.uuid4().hex, _CALLER_OPTIONS[caller]))+ asks: Final = _UPSTREAM_ASKS_FOR_USAGE[setting][caller]+ _assert_one_stop_with(response, _GEMINI_USAGE if asks else _ZERO_USAGE)+ bodies: Final = _model_bodies(wire)+ assert len(bodies) == 1, (caller, bodies)+ assert bodies[0].get("stream_options") == ({"include_usage": True} if asks else None), (caller, bodies[0])+++@pytest.mark.timeout(2 * graceful_stop_seconds() + 120)+@pytest.mark.parametrize("setting", tuple(_SETTING_VALUES))+def test_always_include_stream_usage_decides_what_the_adapter_asks_upstream(+ gateway: Gateway, tmp_path: Path, setting: str+) -> None:+ with ExitStack() as stack:+ wire: Final = stack.enter_context(wire_server(_peer()))+ config: Final = _owned_config(+ tmp_path / f"{setting}.yaml", gateway, wire, {"always_include_stream_usage": _SETTING_VALUES[setting]}+ )+ owned: Final = stack.enter_context(owned_proxy_process(gateway, tmp_path, {}, config=config, workers=2))+ scenario: Final = stack.enter_context(owned.gateway.scenario())+ key: Final = scenario.key(models=[_ADAPTER_MODEL, _NATIVE_MODEL])+ for caller in _CALLER_OPTIONS:+ _assert_caller_option_reaches_upstream(owned.gateway.client, key, wire, setting, caller)+ native: Final = _stream(owned.gateway.client, _NATIVE_MODEL, key, _stream_body(uuid.uuid4().hex, None))+ _assert_one_stop_with(native, _GEMINI_USAGE)+ native_bodies: Final = _model_bodies(wire)+ assert len(native_bodies) == 1, native_bodies+ assert "stream_options" not in native_bodies[0] and "streamOptions" not in native_bodies[0], native_bodies[0]+++def _cache_rows(key: str) -> list[dict[str, JsonValue]]:+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_CACHE_SELECT, (digest,)), lambda rows: len(rows) >= 2, seconds=70)+ return held(lambda: read_rows(_CACHE_SELECT, (digest,)), lambda rows: len(rows) == 2, holding=3)+++def _assert_cache_hit_replays_the_stop_usage(+ client: httpx.Client, key: str, wire: Wire, setting: str, caller: str+) -> None:+ body: Final = _stream_body(uuid.uuid4().hex, _CALLER_OPTIONS[caller])+ usage: Final = _GEMINI_USAGE if _UPSTREAM_ASKS_FOR_USAGE[setting][caller] else _ZERO_USAGE+ _assert_one_stop_with(_stream(client, _ADAPTER_MODEL, key, body), usage)+ _assert_one_stop_with(_stream(client, _ADAPTER_MODEL, key, body), usage)+ assert len(_model_bodies(wire)) == 1, caller+ rows: Final = _cache_rows(key)+ assert [row["cache_hit"] == "True" for row in rows] == [False, True], (caller, rows)+ assert (rows[1]["prompt_tokens"], rows[1]["completion_tokens"]) == (+ rows[0]["prompt_tokens"],+ rows[0]["completion_tokens"],+ ), (caller, rows)+++@pytest.mark.timeout(2 * graceful_stop_seconds() + 120)+@pytest.mark.parametrize("setting", tuple(_SETTING_VALUES))+def test_always_include_stream_usage_cache_hit_replays_the_stop_usage_the_miss_sent(+ gateway: Gateway, tmp_path: Path, setting: str+) -> None:+ with ExitStack() as stack:+ wire: Final = stack.enter_context(wire_server(_peer()))+ config: Final = _owned_config(+ tmp_path / f"{setting}-cache.yaml", gateway, wire, {"always_include_stream_usage": _SETTING_VALUES[setting]}+ )+ owned: Final = stack.enter_context(owned_proxy_process(gateway, tmp_path, {}, config=config, workers=2))+ scenario: Final = stack.enter_context(owned.gateway.scenario())+ for caller in _CALLER_OPTIONS:+ _assert_cache_hit_replays_the_stop_usage(+ owned.gateway.client, scenario.key(models=[_ADAPTER_MODEL]), wire, setting, caller+ )+++def _burst_rows(digests: Sequence[str]) -> list[dict[str, JsonValue]]:+ return read_rows(_BURST_SELECT, (",".join(digests),))+++def _rows_by_key(digests: Sequence[str], expected: int) -> list[dict[str, JsonValue]]:+ eventually(lambda: _burst_rows(digests), lambda rows: len(rows) >= expected, seconds=120)+ return held(lambda: _burst_rows(digests), lambda rows: len(rows) == expected, holding=5)+++def _stream_or_dropped(client: httpx.Client, key: str) -> httpx.Response | None:+ try:+ return _stream(client, _ADAPTER_MODEL, key, _stream_body(uuid.uuid4().hex, None))+ except httpx.TransportError:+ return None+++def _local_port(client: httpx.Client) -> int:+ with client.stream("GET", "/health/liveliness") as response:+ port: Final = int(response.extensions["network_stream"].get_extra_info("client_addr")[1])+ response.read()+ assert response.status_code == 200, response.text+ return port+++def _accepted_client_ports(pid: int, proxy_port: int) -> frozenset[int]:+ return frozenset(+ connection.raddr.port+ for connection in psutil.Process(pid).net_connections(kind="tcp")+ if connection.raddr and connection.laddr.port == proxy_port+ )+++@pytest.mark.timeout(2 * graceful_stop_seconds() + 120)+def test_worker_killed_mid_stream_leaves_the_sibling_billing_each_stream_once(gateway: Gateway, tmp_path: Path) -> None:+ gate: Final = threading.Event()+ with ExitStack() as stack:+ wire: Final = stack.enter_context(wire_server(_peer(gate)))+ stack.callback(gate.set)+ config: Final = _owned_config(tmp_path / "worker-kill.yaml", gateway, wire, {})+ owned: Final = stack.enter_context(owned_proxy_process(gateway, tmp_path, {}, config=config, workers=2))+ workers: Final = eventually(+ lambda: tuple(int(pid) for pid in _STARTED_WORKER.findall(owned.log.read_text())),+ lambda pids: len(pids) == 2,+ seconds=30,+ )+ scenario: Final = stack.enter_context(owned.gateway.scenario())+ proxy_url: Final = owned.gateway.client.base_url+ clients: Final = tuple(+ stack.enter_context(httpx.Client(base_url=str(proxy_url), timeout=60, trust_env=False)) for _ in range(12)+ )+ keys: Final = tuple(scenario.key(models=[_ADAPTER_MODEL]) for _ in clients)+ ports: Final = tuple(_local_port(client) for client in clients)+ pool: Final = stack.enter_context(ThreadPoolExecutor(max_workers=len(clients)))+ futures: Final = tuple(+ pool.submit(_stream_or_dropped, client, key) for client, key in zip(clients, keys, strict=True)+ )+ eventually(wire.received.qsize, lambda size: size >= len(clients), seconds=60)+ shares: Final = {pid: _accepted_client_ports(pid, proxy_url.port or 0) & frozenset(ports) for pid in workers}+ assert sum(map(len, shares.values())) == len(clients), shares+ victim: Final = min((pid for pid in workers if shares[pid]), key=lambda pid: len(shares[pid]))+ with owned.replacing_worker(victim) as kill:+ kill()+ gate.set()+ results: Final = tuple(future.result(timeout=60) for future in futures)+ survivors: Final = tuple(+ (key, result) for port, key, result in zip(ports, keys, results, strict=True) if port not in shares[victim]+ )+ assert survivors, shares+ for _, result in survivors:+ assert result is not None+ _assert_one_stop_with(result, _GEMINI_USAGE)+ second_keys: Final = tuple(scenario.key(models=[_ADAPTER_MODEL]) for _ in range(6))+ second: Final = tuple(_stream_or_dropped(owned.gateway.client, key) for key in second_keys)+ for result in second:+ assert result is not None+ _assert_one_stop_with(result, _GEMINI_USAGE)+ billed: Final = tuple(key for key, _ in survivors) + second_keys+ digests: Final = tuple(sha256(key.encode()).hexdigest() for key in billed)+ rows: Final = _rows_by_key(digests, len(billed))+ assert sorted(str(row["api_key"]) for row in rows) == sorted(digests)+ assert {(row["prompt_tokens"], row["completion_tokens"]) for row in rows} == {+ (_PROMPT_TOKENS, _COMPLETION_TOKENS)+ }, rows+ assert owned.process.poll() is Nonetests/integration/streaming/test_openai_like_stream_usage_wire.pynew file mode 100644index 00000000000..c4ab4794917@@ -0,0 +1,367 @@+import json+import uuid+from collections.abc import Callable+from concurrent.futures import ThreadPoolExecutor+from hashlib import sha256+from typing import Final++import httpx+import pytest+from integration._support.client import Gateway, Scenario, eventually, held+from integration._support.database import read_rows+from integration._support.vertex import service_account_json+from integration._support.wire import Reply, Request, Wire, wire_server+from pydantic import JsonValue, TypeAdapter++_JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])+_PROJECT: Final = "scripted-project"+_PROMPT_TOKENS: Final = 13+_COMPLETION_TOKENS: Final = 4+_CACHED_TOKENS: Final = 5+_REASONING_TOKENS: Final = 2+_USAGE: Final = {+ "prompt_tokens": _PROMPT_TOKENS,+ "completion_tokens": _COMPLETION_TOKENS,+ "total_tokens": _PROMPT_TOKENS + _COMPLETION_TOKENS,+}+_DETAILED_USAGE: Final = {+ **_USAGE,+ "prompt_tokens_details": {"cached_tokens": _CACHED_TOKENS},+ "completion_tokens_details": {"reasoning_tokens": _REASONING_TOKENS},+}+_REPLY_TEXT: Final = "iterator reply"+_SPEND_SELECT: Final = (+ 'SELECT request_id, prompt_tokens, completion_tokens, spend, status FROM "LiteLLM_SpendLogs" WHERE api_key=%s'+)+_CACHE_SELECT: Final = (+ 'SELECT cache_hit, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE api_key=%s ORDER BY "startTime"'+)+_BURST_SELECT: Final = "SELECT api_key, prompt_tokens, completion_tokens FROM \"LiteLLM_SpendLogs\" WHERE api_key = ANY(string_to_array(%s, ','))"+++def _vertex(model: str, location: str) -> Callable[[Gateway, Scenario, Wire], str]:+ def create(gateway: Gateway, scenario: Scenario, wire: Wire) -> str:+ return scenario.model(+ model=f"vertex_ai/{model}",+ api_base=wire.url,+ api_key=None,+ vertex_project=_PROJECT,+ vertex_location=location,+ vertex_credentials=service_account_json(_PROJECT, gateway.upstream_url.rstrip("/")),+ )++ return create+++def _keyed(model: str, **extra: JsonValue) -> Callable[[Gateway, Scenario, Wire], str]:+ def create(_gateway: Gateway, scenario: Scenario, wire: Wire) -> str:+ return scenario.model(model=model, api_base=wire.url, api_key="synthetic-provider-key", **extra)++ return create+++_MEMBERS: Final = {+ "vertex_model_garden": _vertex("xai/grok-4.20-non-reasoning", "global"),+ "vertex_mistral_partner": _vertex("mistral-small-2503", "us-central1"),+ "watsonx": _keyed("watsonx/ibm/granite-3-8b-instruct", zen_api_key="synthetic-zen-key", project_id=_PROJECT),+}+_NON_STREAM_MEMBERS: Final = {+ **_MEMBERS,+ "maritalk": _keyed("maritalk/sabia-3"),+ "amazon_nova": _keyed("amazon_nova/nova-pro-v1"),+}+++def _chunk(choices: list[JsonValue], **extra: JsonValue) -> bytes:+ body: Final = {+ "id": "chatcmpl-iterator",+ "object": "chat.completion.chunk",+ "created": 1700000000,+ "model": "scripted",+ "choices": choices,+ **extra,+ }+ return f"data: {json.dumps(body)}\n\n".encode()+++_DELTA: Final = _chunk([{"index": 0, "delta": {"role": "assistant", "content": _REPLY_TEXT}, "finish_reason": None}])+_FINISH: Final = _chunk([{"index": 0, "delta": {}, "finish_reason": "stop"}])+_USAGE_ONLY: Final = _chunk([], usage=_DETAILED_USAGE)+_DONE: Final = b"data: [DONE]\n\n"+_FINISH_THEN_USAGE: Final = (_DELTA, _FINISH, _USAGE_ONLY, _DONE)+++def _reply_json() -> bytes:+ return json.dumps(+ {+ "id": f"chatcmpl-{uuid.uuid4().hex}",+ "object": "chat.completion",+ "created": 1700000000,+ "model": "scripted",+ "choices": [+ {"index": 0, "message": {"role": "assistant", "content": _REPLY_TEXT}, "finish_reason": "stop"}+ ],+ "usage": _USAGE,+ }+ ).encode()+++def _peer(frames: tuple[bytes, ...]) -> Callable[[Request], Reply]:+ def respond(request: Request) -> Reply:+ if request.method != "POST":+ return Reply(status=404)+ if _JSON_OBJECT.validate_json(request.body).get("stream") is True:+ return Reply(content_type="text/event-stream", chunks=frames)+ return Reply(body=_reply_json())++ return respond+++def _sse_payloads(response: httpx.Response) -> list[dict[str, JsonValue]]:+ lines: Final = [line.removeprefix("data: ") for line in response.text.splitlines() if line.startswith("data: ")]+ return [_JSON_OBJECT.validate_json(line) for line in lines if line != "[DONE]"]+++def _chat_text(chunks: list[dict[str, JsonValue]]) -> str:+ return "".join(+ str(choice.get("delta", {}).get("content") or "")+ for chunk in chunks+ for choice in chunk.get("choices") or [] # comprehension-ok: chunk then choice+ if isinstance(choice, dict) and isinstance(choice.get("delta"), dict)+ )+++def _assert_each_send_billed(key: str, sends: int) -> None:+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_CACHE_SELECT, (digest,)), lambda rows: len(rows) >= sends, seconds=70)+ rows: Final = held(lambda: read_rows(_CACHE_SELECT, (digest,)), lambda found: len(found) == sends, holding=3)+ assert [row["cache_hit"] == "True" for row in rows] == [index > 0 for index in range(sends)], rows+ assert {(row["prompt_tokens"], row["completion_tokens"]) for row in rows} == {+ (_PROMPT_TOKENS, _COMPLETION_TOKENS)+ }, rows+++def _spend_row(key: str) -> dict[str, JsonValue]:+ digest: Final = sha256(key.encode()).hexdigest()+ eventually(lambda: read_rows(_SPEND_SELECT, (digest,)), lambda rows: len(rows) >= 1, seconds=70)+ rows: Final = held(lambda: read_rows(_SPEND_SELECT, (digest,)), lambda found: len(found) == 1, holding=3)+ return rows[0]+++def _stream_chat(gateway: Gateway, model: str, key: str) -> httpx.Response:+ return _stream_chat_about(gateway, model, key, uuid.uuid4().hex)+++def _stream_chat_about(gateway: Gateway, model: str, key: str, marker: str) -> httpx.Response:+ return gateway.client.post(+ "/v1/chat/completions",+ json={+ "model": model,+ "messages": [{"role": "user", "content": f"iterator probe {marker}"}],+ "stream": True,+ "stream_options": {"include_usage": True},+ },+ headers={"Authorization": f"Bearer {key}"},+ )+++@pytest.mark.parametrize("member", tuple(_MEMBERS))+def test_chat_stream_keeps_the_usage_chunk_sent_after_the_finish_chunk(gateway: Gateway, member: str) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS[member](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_chat(gateway, model, key)+ assert response.status_code == 200, response.text+ chunks: Final = _sse_payloads(response)+ assert _chat_text(chunks) == _REPLY_TEXT, response.text+ usage: Final = [chunk["usage"] for chunk in chunks if chunk.get("usage")]+ assert len(usage) == 1, response.text+ assert {name: usage[0][name] for name in _USAGE} == _USAGE, response.text+ row: Final = _spend_row(key)+ assert (row["prompt_tokens"], row["completion_tokens"]) == (_PROMPT_TOKENS, _COMPLETION_TOKENS), row+ assert len(wire.drain()) == 1+++@pytest.mark.parametrize("member", tuple(_MEMBERS))+def test_chat_stream_usage_carries_the_upstream_token_details(gateway: Gateway, member: str) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS[member](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_chat(gateway, model, key)+ assert response.status_code == 200, response.text+ usage: Final = [chunk["usage"] for chunk in _sse_payloads(response) if chunk.get("usage")]+ assert len(usage) == 1, response.text+ details: Final = usage[0]+ assert isinstance(details, dict)+ assert details["prompt_tokens_details"]["cached_tokens"] == _CACHED_TOKENS, response.text+ assert details["completion_tokens_details"]["reasoning_tokens"] == _REASONING_TOKENS, response.text+ assert len(wire.drain()) == 1+++@pytest.mark.parametrize("member", tuple(_MEMBERS))+def test_chat_stream_cache_hit_replays_the_usage_chunk(gateway: Gateway, member: str) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS[member](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ marker: Final = uuid.uuid4().hex+ responses: Final = tuple(_stream_chat_about(gateway, model, key, marker) for _ in range(2))+ for response in responses:+ assert response.status_code == 200, response.text+ chunks = _sse_payloads(response)+ assert _chat_text(chunks) == _REPLY_TEXT, response.text+ usage = [chunk["usage"] for chunk in chunks if chunk.get("usage")]+ assert len(usage) == 1, response.text+ assert {name: usage[0][name] for name in _USAGE} == _USAGE, response.text+ assert len(wire.drain()) == 1+ _assert_each_send_billed(key, 2)+++@pytest.mark.parametrize("member", tuple(_NON_STREAM_MEMBERS))+def test_chat_non_stream_usage_is_unchanged(gateway: Gateway, member: str) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ model: Final = _NON_STREAM_MEMBERS[member](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = gateway.request(+ "POST", "/v1/chat/completions", {"model": model, "messages": [{"role": "user", "content": "hi"}]}, key=key+ )+ assert response.status_code == 200, response.text+ body: Final = _JSON_OBJECT.validate_json(response.content)+ assert {name: body["usage"][name] for name in _USAGE} == _USAGE, response.text+ row: Final = _spend_row(key)+ assert (row["prompt_tokens"], row["completion_tokens"]) == (_PROMPT_TOKENS, _COMPLETION_TOKENS), row+ assert len(wire.drain()) == 1+++@pytest.mark.parametrize("sends", [1, 2], ids=["miss", "cache_hit"])+def test_messages_stream_bridge_keeps_usage_after_finish(gateway: Gateway, sends: int) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS["vertex_model_garden"](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ body: Final = {+ "model": model,+ "max_tokens": 64,+ "messages": [{"role": "user", "content": f"bridge probe {uuid.uuid4().hex}"}],+ "stream": True,+ }+ responses: Final = tuple(+ gateway.client.post("/v1/messages", json=body, headers={"Authorization": f"Bearer {key}"})+ for _ in range(sends)+ )+ for response in responses:+ assert response.status_code == 200, response.text+ deltas = [event for event in _sse_payloads(response) if event.get("type") == "message_delta"]+ assert len(deltas) == 1, response.text+ assert deltas[0]["usage"]["output_tokens"] == _COMPLETION_TOKENS, response.text+ _assert_each_send_billed(key, sends)+ assert len(wire.drain()) == 1+++@pytest.mark.parametrize("sends", [1, 2], ids=["miss", "cache_hit"])+def test_responses_stream_bridge_keeps_usage_after_finish(gateway: Gateway, sends: int) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS["vertex_model_garden"](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ body: Final = {"model": model, "input": f"bridge probe {uuid.uuid4().hex}", "stream": True}+ responses: Final = tuple(+ gateway.client.post("/v1/responses", json=body, headers={"Authorization": f"Bearer {key}"})+ for _ in range(sends)+ )+ for response in responses:+ assert response.status_code == 200, response.text+ completed = [event for event in _sse_payloads(response) if event.get("type") == "response.completed"]+ assert len(completed) == 1, response.text+ usage = completed[0]["response"]["usage"]+ assert (usage["input_tokens"], usage["output_tokens"]) == (_PROMPT_TOKENS, _COMPLETION_TOKENS), (+ response.text+ )+ _assert_each_send_billed(key, sends)+ assert len(wire.drain()) == 1+++@pytest.mark.parametrize(+ "frames",+ [+ pytest.param((_DELTA, b"data: {not json\n\n", _FINISH, _USAGE_ONLY, _DONE), id="malformed_data_line"),+ pytest.param((_DELTA, b"data: \n\n", _FINISH, _USAGE_ONLY, _DONE), id="blank_data_line"),+ pytest.param((_DELTA, _FINISH, b"\n" * 5000, _USAGE_ONLY, _DONE), id="five_kb_of_blank_lines"),+ pytest.param(+ (b"id: 1\nevent: message\n" + _DELTA, b"id: 2\nevent: message\n" + _FINISH, _USAGE_ONLY, _DONE),+ id="event_and_id_lines",+ ),+ ],+)+def test_chat_stream_tolerates_noise_between_chunks(gateway: Gateway, frames: tuple[bytes, ...]) -> None:+ with wire_server(_peer(frames)) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS["vertex_model_garden"](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_chat(gateway, model, key)+ assert response.status_code == 200, response.text+ chunks: Final = _sse_payloads(response)+ assert _chat_text(chunks) == _REPLY_TEXT, response.text+ usage: Final = [chunk["usage"] for chunk in chunks if chunk.get("usage")]+ assert len(usage) == 1 and usage[0]["prompt_tokens"] == _PROMPT_TOKENS, response.text+ assert len(wire.drain()) == 1+++def test_chat_stream_of_only_blank_lines_ends_cleanly(gateway: Gateway) -> None:+ with wire_server(_peer((b"\n\n", b"\n\n", b"\n\n"))) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS["vertex_model_garden"](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_chat(gateway, model, key)+ assert response.status_code == 200, response.text+ assert _chat_text(_sse_payloads(response)) == "", response.text+ assert "data: [DONE]" in response.text, response.text+ assert len(wire.drain()) == 1+ assert gateway.request("GET", "/health/liveliness").status_code == 200+++def test_chat_stream_upstream_drop_after_finish_ends_the_stream(gateway: Gateway) -> None:+ def respond(request: Request) -> Reply:+ if _JSON_OBJECT.validate_json(request.body).get("stream") is not True:+ return Reply(body=_reply_json())+ return Reply(+ content_type="text/event-stream", chunks=_FINISH_THEN_USAGE, abort_after=2, pause_between_chunks=1.5+ )++ with wire_server(respond) as wire, gateway.scenario() as scenario:+ model: Final = _MEMBERS["vertex_model_garden"](gateway, scenario, wire)+ key: Final = scenario.key(models=[model])+ response: Final = _stream_chat(gateway, model, key)+ assert response.status_code == 200, response.text+ assert _chat_text(_sse_payloads(response)) == _REPLY_TEXT, response.text+ assert len(wire.drain()) == 1+ assert gateway.request("GET", "/health/liveliness").status_code == 200+ other: Final = scenario.key(models=[model])+ assert (+ gateway.request(+ "POST",+ "/v1/chat/completions",+ {"model": model, "messages": [{"role": "user", "content": "x"}]},+ key=other,+ ).status_code+ == 200+ )+++def _burst_rows(digests: tuple[str, ...]) -> list[dict[str, JsonValue]]:+ return read_rows(_BURST_SELECT, (",".join(digests),))+++def test_chat_stream_burst_bills_every_request_once(gateway: Gateway) -> None:+ with wire_server(_peer(_FINISH_THEN_USAGE)) as wire, gateway.scenario() as scenario:+ models: Final = {name: create(gateway, scenario, wire) for name, create in _MEMBERS.items()}+ names: Final = tuple(_MEMBERS) * 10+ keys: Final = tuple(scenario.key(models=[models[name]]) for name in names)+ with ThreadPoolExecutor(max_workers=len(keys)) as pool:+ responses: Final = tuple(+ pool.map(lambda pair: _stream_chat(gateway, models[pair[0]], pair[1]), zip(names, keys, strict=True))+ )+ assert [response.status_code for response in responses] == [200] * len(keys)+ digests: Final = tuple(sha256(key.encode()).hexdigest() for key in keys)+ eventually(lambda: _burst_rows(digests), lambda rows: len(rows) >= len(keys), seconds=90)+ rows: Final = held(lambda: _burst_rows(digests), lambda found: len(found) == len(keys), holding=3)+ assert sorted(str(row["api_key"]) for row in rows) == sorted(digests)+ assert {(row["prompt_tokens"], row["completion_tokens"]) for row in rows} == {+ (_PROMPT_TOKENS, _COMPLETION_TOKENS)+ }, rows+ assert len(wire.drain()) == len(keys)tests/unit/google_genai/test_google_genai_handler.pyindex 68f24c1c0f5..91d4f542a13 100644@@ -3,12 +3,22 @@ Test to verify the Google GenAI generate_content handler functionality """ +import asyncio+import json+import time from unittest.mock import AsyncMock, MagicMock, patch+from typing import Final +import httpx import pytest+import respx++import litellm+from litellm.integrations.custom_logger import CustomLogger from litellm.google_genai.adapters.handler import GenerateContentToCompletionHandler-from litellm.google_genai.adapters.transformation import GoogleGenAIAdapter+from litellm.google_genai.adapters.transformation import GoogleGenAIAdapter, GoogleGenAIStreamWrapper+from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices, Usage def test_stream_response_when_stream_requested_sync():@@ -275,3 +285,329 @@ async def test_generate_content_adapter_preserves_proxy_server_request(): assert "proxy_server_request" in called_kwargs, "proxy_server_request was dropped from completion_kwargs" assert called_kwargs["proxy_server_request"] == dummy_proxy_request+++_CHAT_SSE = (+ 'data: {"id":"c","object":"chat.completion.chunk","created":1,"model":"m",'+ '"choices":[{"index":0,"delta":{"role":"assistant","content":"144"},"finish_reason":null}]}\n\n'+ 'data: {"id":"c","object":"chat.completion.chunk","created":1,"model":"m",'+ '"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}\n\n'+ "data: [DONE]\n\n"+)+_CHAT_BODY: dict[str, object] = {+ "id": "c",+ "object": "chat.completion",+ "created": 1,+ "model": "m",+ "choices": [{"index": 0, "message": {"role": "assistant", "content": "144"}, "finish_reason": "stop"}],+ "usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4},+}+++_CHAT_SSE_WITH_USAGE = _CHAT_SSE.replace(+ "data: [DONE]",+ 'data: {"id":"c","object":"chat.completion.chunk","created":1,"model":"m","choices":[],'+ '"usage":{"prompt_tokens":340,"completion_tokens":1,"total_tokens":341}}\n\n'+ "data: [DONE]",+)++_ANTHROPIC_SSE: Final = "".join(+ f"event: {event}\ndata: {json.dumps(data)}\n\n"+ for event, data in (+ (+ "message_start",+ {+ "type": "message_start",+ "message": {+ "id": "msg_1",+ "type": "message",+ "role": "assistant",+ "model": "claude-haiku-4-5",+ "content": [],+ "stop_reason": None,+ "usage": {"input_tokens": 20, "output_tokens": 1},+ },+ },+ ),+ (+ "content_block_start",+ {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},+ ),+ (+ "content_block_delta",+ {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "144"}},+ ),+ ("content_block_stop", {"type": "content_block_stop", "index": 0}),+ (+ "message_delta",+ {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"output_tokens": 5}},+ ),+ ("message_stop", {"type": "message_stop"}),+ )+)+++class _FailureCallTypes(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self.call_types: list[str] = []++ def log_failure_event(self, kwargs, response_obj, start_time, end_time):+ self.call_types.append(kwargs["call_type"])++ async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):+ self.call_types.append(kwargs["call_type"])+++class _SuccessPayloads(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self.model_groups: list[str | None] = []++ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):+ self.model_groups.append(kwargs["standard_logging_object"]["model_group"])+++@pytest.fixture+def _httpx_transport(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)+++async def _drain(stream: object) -> None:+ async for _ in stream:+ pass+++@pytest.mark.asyncio+async def test_generate_content_adapter_keeps_provider_error_status(_httpx_transport: None):+ from litellm.types.router import GenericLiteLLMParams++ with respx.mock() as upstream:+ upstream.post("http://upstream.test/v1/chat/completions").mock(+ return_value=httpx.Response(400, json={"error": {"message": "max_tokens must be positive"}})+ )+ with pytest.raises(litellm.BadRequestError) as exc_info:+ await GenerateContentToCompletionHandler.async_generate_content_handler(+ model="hosted_vllm/grok",+ contents=[{"role": "user", "parts": [{"text": "Hello"}]}],+ litellm_params=GenericLiteLLMParams(api_base="http://upstream.test/v1", api_key="k"),+ config={"maxOutputTokens": -5},+ )++ assert exc_info.value.status_code == 400+++@pytest.mark.asyncio+async def test_generate_content_adapter_logs_router_model_group(+ _httpx_transport: None, monkeypatch: pytest.MonkeyPatch+):+ from litellm.types.router import GenericLiteLLMParams++ success_payloads = _SuccessPayloads()+ monkeypatch.setattr(litellm, "callbacks", [success_payloads])++ with respx.mock() as upstream:+ upstream.post("http://upstream.test/v1/chat/completions").mock(+ return_value=httpx.Response(200, json=_CHAT_BODY)+ )+ await GenerateContentToCompletionHandler.async_generate_content_handler(+ model="hosted_vllm/grok",+ contents=[{"role": "user", "parts": [{"text": "Hello"}]}],+ litellm_params=GenericLiteLLMParams(api_base="http://upstream.test/v1", api_key="k"),+ litellm_metadata={"model_group": "grok-group"},+ )+ await asyncio.sleep(0.5)++ assert "grok-group" in success_payloads.model_groups+++@pytest.mark.parametrize(+ ("stream", "requested_stream_options", "expected_stream_options"),+ [+ (True, None, {"include_usage": True}),+ (True, {"include_usage": True}, {"include_usage": True}),+ (True, {"include_usage": False}, None),+ (False, None, None),+ ],+)+@pytest.mark.asyncio+async def test_generate_content_adapter_asks_openai_compatible_upstream_for_stream_usage(+ _httpx_transport: None,+ stream: bool,+ requested_stream_options: dict[str, bool] | None,+ expected_stream_options: dict[str, bool] | None,+):+ from litellm.types.router import GenericLiteLLMParams++ with respx.mock() as upstream:+ route = upstream.post("http://upstream.test/v1/chat/completions").mock(+ return_value=(+ httpx.Response(200, text=_CHAT_SSE, headers={"content-type": "text/event-stream"})+ if stream+ else httpx.Response(200, json=_CHAT_BODY)+ )+ )+ response = await GenerateContentToCompletionHandler.async_generate_content_handler(+ model="hosted_vllm/grok",+ contents=[{"role": "user", "parts": [{"text": "Hello"}]}],+ litellm_params=GenericLiteLLMParams(api_base="http://upstream.test/v1", api_key="k"),+ stream=stream,+ **({} if requested_stream_options is None else {"stream_options": requested_stream_options}),+ )+ if stream:+ await _drain(response)++ assert json.loads(route.calls[0].request.content).get("stream_options") == expected_stream_options+++@pytest.mark.parametrize(+ ("model", "api_base", "upstream_url", "upstream_sse", "prompt_tokens"),+ [+ (+ "hosted_vllm/grok",+ "http://upstream.test/v1",+ "http://upstream.test/v1/chat/completions",+ _CHAT_SSE_WITH_USAGE,+ 340,+ ),+ ("anthropic/claude-haiku-4-5", "http://upstream.test", "http://upstream.test/v1/messages", _ANTHROPIC_SSE, 20),+ ],+ ids=["openai_compatible", "anthropic"],+)+@pytest.mark.asyncio+async def test_generate_content_adapter_stream_ends_with_upstream_usage(+ _httpx_transport: None, model: str, api_base: str, upstream_url: str, upstream_sse: str, prompt_tokens: int+):+ payloads: Final = await _adapter_stream_payloads(model, api_base, upstream_url, upstream_sse)++ assert payloads[-1]["candidates"][0]["finishReason"] == "STOP"+ assert payloads[-1]["usageMetadata"]["promptTokenCount"] == prompt_tokens+++@pytest.mark.parametrize(+ ("model", "api_base", "upstream_url", "upstream_sse"),+ [+ ("hosted_vllm/grok", "http://upstream.test/v1", "http://upstream.test/v1/chat/completions", _CHAT_SSE),+ ("anthropic/claude-haiku-4-5", "http://upstream.test", "http://upstream.test/v1/messages", _ANTHROPIC_SSE),+ ],+ ids=["openai_compatible", "anthropic"],+)+@pytest.mark.asyncio+async def test_generate_content_adapter_stream_without_usage_still_ends_with_stop(+ _httpx_transport: None, model: str, api_base: str, upstream_url: str, upstream_sse: str+):+ payloads: Final = await _adapter_stream_payloads(+ model, api_base, upstream_url, upstream_sse, stream_options={"include_usage": False}+ )++ assert [payload["candidates"][0].get("finishReason") for payload in payloads][-1] == "STOP"+++async def _adapter_stream_payloads(+ model: str, api_base: str, upstream_url: str, upstream_sse: str, **kwargs: object+):+ from litellm.types.router import GenericLiteLLMParams++ with respx.mock() as upstream:+ upstream.post(upstream_url).mock(+ return_value=httpx.Response(200, text=upstream_sse, headers={"content-type": "text/event-stream"})+ )+ response = await GenerateContentToCompletionHandler.async_generate_content_handler(+ model=model,+ contents=[{"role": "user", "parts": [{"text": "Hello"}]}],+ litellm_params=GenericLiteLLMParams(api_base=api_base, api_key="k"),+ stream=True,+ **kwargs,+ )+ events: Final = [chunk async for chunk in response]++ return [+ json.loads(line.removeprefix("data: "))+ for line in b"".join(events).decode().splitlines()+ if line.startswith("data: {")+ ]+++@pytest.mark.parametrize("entrypoint", ["agenerate_content", "agenerate_content_stream"])+@pytest.mark.asyncio+async def test_generate_content_adapter_failure_is_logged_once(+ _httpx_transport: None, monkeypatch: pytest.MonkeyPatch, entrypoint: str+):+ failures = _FailureCallTypes()+ monkeypatch.setattr(litellm, "callbacks", [failures])++ with respx.mock() as upstream:+ upstream.post("http://upstream.test/v1/chat/completions").mock(+ return_value=httpx.Response(400, json={"error": {"message": "max_tokens must be positive"}})+ )+ with pytest.raises(litellm.BadRequestError):+ await getattr(litellm.google_genai, entrypoint)(+ model="hosted_vllm/grok",+ contents=[{"role": "user", "parts": [{"text": "Hello"}]}],+ api_base="http://upstream.test/v1",+ api_key="k",+ )+ await asyncio.sleep(0.5)++ assert failures.call_types == ["acompletion"]+++def test_sync_generate_content_adapter_failure_is_logged_once(_httpx_transport: None, monkeypatch: pytest.MonkeyPatch):+ failures = _FailureCallTypes()+ monkeypatch.setattr(litellm, "callbacks", [failures])++ with respx.mock() as upstream:+ upstream.post("http://upstream.test/v1/chat/completions").mock(+ return_value=httpx.Response(400, json={"error": {"message": "max_tokens must be positive"}})+ )+ with pytest.raises(litellm.BadRequestError):+ litellm.google_genai.generate_content(+ model="hosted_vllm/grok",+ contents=[{"role": "user", "parts": [{"text": "Hello"}]}],+ api_base="http://upstream.test/v1",+ api_key="k",+ )+ time.sleep(0.5)++ assert failures.call_types == ["completion"]+++def _stream_chunk(content: str | None, finish_reason: str | None, usage: Usage | None = None) -> ModelResponseStream:+ return ModelResponseStream(+ choices=[StreamingChoices(index=0, delta=Delta(content=content), finish_reason=finish_reason)],+ **({} if usage is None else {"usage": usage}),+ )+++_FINISH_WITHOUT_USAGE: Final = (_stream_chunk("144", None), _stream_chunk(None, "stop"))+_FINISH_WITH_USAGE: Final = (+ _stream_chunk("144", None),+ _stream_chunk(None, "stop", Usage(prompt_tokens=20, completion_tokens=1, total_tokens=21)),+)+++@pytest.mark.parametrize(+ ("chunks", "prompt_tokens"), [(_FINISH_WITHOUT_USAGE, 0), (_FINISH_WITH_USAGE, 20)], ids=["no_usage", "usage"]+)+def test_stream_wrapper_iteration_ends_with_one_stop_event(chunks: tuple[ModelResponseStream, ...], prompt_tokens: int):+ events: Final = list(GoogleGenAIStreamWrapper(completion_stream=iter(chunks)))++ assert [event["candidates"][0].get("finishReason") for event in events] == [None, "STOP"]+ assert events[-1]["usageMetadata"]["promptTokenCount"] == prompt_tokens+++@pytest.mark.parametrize(+ ("chunks", "prompt_tokens"), [(_FINISH_WITHOUT_USAGE, 0), (_FINISH_WITH_USAGE, 20)], ids=["no_usage", "usage"]+)+@pytest.mark.asyncio+async def test_stream_wrapper_async_iteration_ends_with_one_stop_event(+ chunks: tuple[ModelResponseStream, ...], prompt_tokens: int+):+ async def completion_stream():+ for chunk in chunks:+ yield chunk++ events: Final = [event async for event in GoogleGenAIStreamWrapper(completion_stream=completion_stream())]++ assert [event["candidates"][0].get("finishReason") for event in events] == [None, "STOP"]+ assert events[-1]["usageMetadata"]["promptTokenCount"] == prompt_tokenstests/unit/google_genai/test_google_genai_main.pyindex 238fff7deca..39ca62d3935 100644@@ -259,3 +259,29 @@ async def test_native_fields_forwarded_on_async_stream(): body = mock_post.call_args.kwargs["json"] assert body["safetySettings"] == safety_settings assert "safetySettings" not in body.get("generationConfig", {})+++@pytest.mark.parametrize("base_model, uses_native_api", [("gemini-2.5-flash", True), (None, False)])+def test_vertex_generate_content_routes_by_deployment_base_model(base_model, uses_native_api):+ import datetime++ from litellm.google_genai.main import GenerateContentHelper+ from litellm.litellm_core_utils.litellm_logging import Logging++ logging_obj = Logging(+ model="vertex_ai/openai/tuned-endpoint",+ messages=[],+ stream=False,+ call_type="agenerate_content",+ start_time=datetime.datetime.now(),+ litellm_call_id="route-test",+ function_id="route-test",+ )+ setup = GenerateContentHelper.setup_generate_content_call(+ model="vertex_ai/openai/tuned-endpoint",+ contents=[{"role": "user", "parts": [{"text": "Say hi"}]}],+ litellm_logging_obj=logging_obj,+ **({} if base_model is None else {"base_model": base_model}),+ )++ assert (setup.generate_content_provider_config is not None) is uses_native_apitests/unit/llms/databricks/test_databricks_streaming_utils.pyindex 5612864a841..5c5527f7be8 100644@@ -7,6 +7,14 @@ set. `chunk_parser` previously did `choices[0]` unconditionally, raising `IndexError` -> `MidStreamFallbackError` and crashing the stream. """ +import json+from typing import Final++import httpx+import pytest+import respx++import litellm from litellm.llms.databricks.streaming_utils import ModelResponseIterator @@ -62,3 +70,155 @@ def test_chunk_parser_normal_content_chunk_still_works(): result = iterator.chunk_parser(chunk=chunk) assert result["text"] == "hi"+++def test_chunk_parser_keeps_cached_tokens_on_usage_only_chunk():+ iterator = ModelResponseIterator(streaming_response=None, sync_stream=True)+ usage_only_chunk = {+ "id": "chatcmpl-x",+ "object": "chat.completion.chunk",+ "created": 1,+ "model": "m",+ "choices": [],+ "usage": {+ "prompt_tokens": 340,+ "completion_tokens": 1,+ "total_tokens": 341,+ "prompt_tokens_details": {"cached_tokens": 320},+ },+ }++ result = iterator.chunk_parser(chunk=usage_only_chunk)++ assert result["usage"] is not None+ assert result["usage"]["prompt_tokens_details"] is not None+ assert result["usage"]["prompt_tokens_details"]["cached_tokens"] == 320+++def _sse_event(body: dict[str, object]) -> bytes:+ return f"data: {json.dumps(body)}\n\n".encode()+++@pytest.mark.respx(assert_all_called=True)+def test_stream_usage_after_finish_chunk_reaches_the_caller(respx_mock: respx.MockRouter) -> None:+ chunk_header: Final = {"id": "chatcmpl-x", "object": "chat.completion.chunk", "created": 1, "model": "sabia-3"}+ stream_body: Final = b"".join(+ (+ _sse_event({**chunk_header, "choices": [{"index": 0, "delta": {"role": "assistant", "content": "144"}}]}),+ _sse_event({**chunk_header, "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}),+ _sse_event(+ {+ **chunk_header,+ "choices": [],+ "usage": {+ "prompt_tokens": 340,+ "completion_tokens": 1,+ "total_tokens": 341,+ "prompt_tokens_details": {"cached_tokens": 320},+ },+ }+ ),+ b"data: [DONE]\n\n",+ )+ )+ respx_mock.post("http://upstream.test/chat/completions").mock(+ return_value=httpx.Response(200, headers={"content-type": "text/event-stream"}, content=stream_body)+ )++ stream: Final = litellm.completion(+ model="maritalk/sabia-3",+ api_base="http://upstream.test",+ api_key="sk-test",+ messages=[{"role": "user", "content": "What is 12*12?"}],+ stream=True,+ stream_options={"include_usage": True},+ )+ usages: Final = [chunk.usage for chunk in stream if getattr(chunk, "usage", None)]++ assert [(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens) for usage in usages] == [(340, 320)]+++def test_chunk_parser_keeps_usage_on_the_finish_chunk():+ iterator: Final = ModelResponseIterator(streaming_response=None, sync_stream=True)++ result: Final = iterator.chunk_parser(+ chunk={+ "id": "chatcmpl-x",+ "object": "chat.completion.chunk",+ "created": 1,+ "model": "m",+ "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],+ "usage": {"prompt_tokens": 340, "completion_tokens": 1, "total_tokens": 341},+ }+ )++ assert result["is_finished"] is True+ assert result["usage"] is not None+ assert result["usage"]["prompt_tokens"] == 340+++_SSE_LINES_WITH_BLANK_SEPARATORS: Final = (+ 'data: {"id":"c","object":"chat.completion.chunk","created":1,"model":"m",'+ '"choices":[{"index":0,"delta":{"content":"144"},"finish_reason":null}]}',+ "",+ 'data: {"id":"c","object":"chat.completion.chunk","created":1,"model":"m",'+ '"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}',+ "",+ "data: {not json",+ "",+ 'data: {"id":"c","object":"chat.completion.chunk","created":1,"model":"m","choices":[],'+ '"usage":{"prompt_tokens":340,"completion_tokens":1,"total_tokens":341}}',+ "",+ "data: [DONE]",+ "",+)+++@pytest.mark.asyncio+async def test_async_stream_skips_blank_sse_lines_and_keeps_the_usage_chunk():+ async def lines():+ for line in _SSE_LINES_WITH_BLANK_SEPARATORS:+ yield line++ chunks: Final = [chunk async for chunk in ModelResponseIterator(streaming_response=lines(), sync_stream=False)]++ assert [(chunk["text"], chunk["finish_reason"], (chunk["usage"] or {}).get("prompt_tokens")) for chunk in chunks] == [+ ("144", "", None),+ ("", "stop", None),+ ("", "", None),+ ("", "", 340),+ ]+++class _BrokenTransport(ValueError):+ pass+++def test_sync_stream_transport_error_surfaces_as_runtime_error():+ def lines():+ yield _SSE_LINES_WITH_BLANK_SEPARATORS[0]+ raise _BrokenTransport("connection reset")++ with pytest.raises(RuntimeError, match="connection reset"):+ list(ModelResponseIterator(streaming_response=lines(), sync_stream=True))+++@pytest.mark.asyncio+async def test_async_stream_transport_error_surfaces_as_runtime_error():+ async def lines():+ yield _SSE_LINES_WITH_BLANK_SEPARATORS[0]+ raise _BrokenTransport("connection reset")++ with pytest.raises(RuntimeError, match="connection reset"):+ [chunk async for chunk in ModelResponseIterator(streaming_response=lines(), sync_stream=False)]+++@pytest.mark.asyncio+async def test_async_stream_without_done_marker_ends_after_the_last_chunk():+ async def lines():+ for line in _SSE_LINES_WITH_BLANK_SEPARATORS[:4]:+ yield line++ chunks: Final = [chunk async for chunk in ModelResponseIterator(streaming_response=lines(), sync_stream=False)]++ assert [(chunk["text"], chunk["finish_reason"]) for chunk in chunks] == [("144", ""), ("", "stop")]tests/unit/llms/openai/test_openai.pyindex 713b331f8a5..e37fd4b6ae5 100644@@ -464,6 +464,22 @@ def _openai_sse_chunk(chunk_id: str, model: str, choices: list[JsonValue]) -> by return f"data: {json.dumps(body)}\n\n".encode() +def _openai_sse_usage_chunk(chunk_id: str, model: str, *, prompt_tokens: int, completion_tokens: int) -> bytes:+ body: Final[dict[str, JsonValue]] = {+ "id": chunk_id,+ "object": "chat.completion.chunk",+ "created": 0,+ "model": model,+ "choices": [],+ "usage": {+ "prompt_tokens": prompt_tokens,+ "completion_tokens": completion_tokens,+ "total_tokens": prompt_tokens + completion_tokens,+ },+ }+ return f"data: {json.dumps(body)}\n\n".encode()++ @pytest.mark.respx(assert_all_called=True) @pytest.mark.parametrize("model", ["o1", "o3-mini"]) def test_o1_parallel_tool_calls_are_dropped_for_unsupported_models(respx_mock: respx.MockRouter, model: str) -> None:@@ -684,6 +700,7 @@ async def test_openai_via_gemini_streaming_bridge_translates_openai_chunks( [{"index": 0, "delta": {"content": "crossed the sky."}, "finish_reason": None}], ), _openai_sse_chunk("chatcmpl-bridge", "gpt-3.5-turbo", [{"index": 0, "delta": {}, "finish_reason": "stop"}]),+ _openai_sse_usage_chunk("chatcmpl-bridge", "gpt-3.5-turbo", prompt_tokens=12, completion_tokens=6), b"data: [DONE]\n\n", ) )@@ -738,7 +755,7 @@ async def test_openai_via_gemini_streaming_bridge_translates_openai_chunks( "candidates": [ {"content": {"parts": [], "role": "model"}, "finishReason": "STOP", "index": 0, "safetyRatings": []} ],- "usageMetadata": {"promptTokenCount": 0, "candidatesTokenCount": 0, "totalTokenCount": 0},+ "usageMetadata": {"promptTokenCount": 12, "candidatesTokenCount": 6, "totalTokenCount": 18}, }, ) tests/unit/llms/vertex_ai/test_vertex_ai_common_utils.pyindex f7ebc3eab54..304424b2acc 100644@@ -1942,3 +1942,35 @@ def test_convert_vertex_datetime_to_openai_datetime_is_the_utc_epoch_on_any_host vertex_datetime: str, utc_epoch: int ) -> None: assert convert_vertex_datetime_to_openai_datetime(vertex_datetime) == utc_epoch+++@pytest.mark.parametrize(+ ("base_model", "vertex_counted_tokens"),+ [(None, None), ("gemini-3.8-flash", 7)],+ ids=["model_garden_route", "gemini_base_model"],+)+@pytest.mark.asyncio+async def test_vertex_ai_token_counter_routes_like_generate_content(+ base_model: str | None, vertex_counted_tokens: int | None+):+ from litellm.llms.vertex_ai.common_utils import VertexAITokenCounter++ with patch(+ "litellm.llms.vertex_ai.count_tokens.handler.VertexAITokenCounter.acount_tokens",+ new=AsyncMock(return_value={"totalTokens": 7}),+ ):+ result: Final = await VertexAITokenCounter().count_tokens(+ model_to_use="xai/grok-4.20-non-reasoning",+ messages=[{"role": "user", "content": "Hello"}],+ contents=None,+ deployment={+ "litellm_params": {+ "vertex_project": "test-project",+ "vertex_location": "global",+ **({} if base_model is None else {"base_model": base_model}),+ }+ },+ request_model="vertex_ai/xai/grok-4.20-non-reasoning",+ )++ assert (None if result is None else result.total_tokens) == vertex_counted_tokenstests/unit/proxy/test_common_request_processing.pyindex 09ed6399d68..bf6eb062507 100644@@ -8295,6 +8295,32 @@ def _apply_stream_usage_tracking( class TestApplyStreamUsageTracking:+ @pytest.mark.parametrize(+ ("general_settings", "requested_stream_options", "expected_stream_options"),+ [+ ({"always_include_stream_usage": False}, None, {"include_usage": False}),+ ({}, None, None),+ ({"always_include_stream_usage": False}, {"include_usage": True}, {"include_usage": True}),+ ],+ )+ def test_generate_content_stream_passes_only_the_admin_opt_out(+ self,+ general_settings: dict[str, bool],+ requested_stream_options: dict[str, bool] | None,+ expected_stream_options: dict[str, bool] | None,+ ):+ data = {+ "stream": True,+ "model": "grok-4.20-non-reasoning",+ **({} if requested_stream_options is None else {"stream_options": requested_stream_options}),+ }++ _apply_stream_usage_tracking(+ data=data, general_settings=general_settings, route_type="agenerate_content_stream"+ )++ assert data.get("stream_options") == expected_stream_options+ def test_default_injects_usage_and_marks_strip_for_chat_completions(self): data = {"stream": True, "model": "gpt-5.4-nano"} tests/unit/test_cost_calculator.pyindex f58e9672c3b..3db4432bd6c 100644@@ -9222,6 +9222,28 @@ def test_live_session_uses_latest_snapshot_when_closed_usage_is_missing(_local_m assert cost == pytest.approx(15 * rate) +def test_vertex_partner_model_cost_keeps_its_publisher_prefix(_local_model_cost_map: None) -> None:+ entry: Final = litellm.model_cost["vertex_ai/xai/grok-4.20-non-reasoning"]+ usage: Final = Usage(+ prompt_tokens=340,+ completion_tokens=1,+ total_tokens=341,+ prompt_tokens_details=PromptTokensDetailsWrapper(cached_tokens=320),+ )+ response: Final = ModelResponse(model="xai/grok-4.20-non-reasoning", usage=usage)++ cost: Final = completion_cost(+ completion_response=response, model="xai/grok-4.20-non-reasoning", custom_llm_provider="vertex_ai"+ )++ expected: Final = (+ 20 * entry["input_cost_per_token"]+ + 320 * entry["cache_read_input_token_cost"]+ + 1 * entry["output_cost_per_token"]+ )+ assert cost == pytest.approx(expected)++ @pytest.mark.parametrize("query, expected_queries", [("litellm release notes", 1), (["a", "b", "c"], 3)]) def test_completion_cost_search_uses_search_tool_input_cost_per_query( _local_model_cost_map: None, query: str | list[str], expected_queries: inttests/unit/test_utils.pyindex 768f8b7bf59..98025f3a79a 100644@@ -7009,6 +7009,32 @@ def test_get_model_info_gemini(monkeypatch): assert info.get("rpm") is not None, f"{model} does not have rpm" +@pytest.mark.parametrize(+ "model, litellm_params, expects_native_config",+ [+ ("gemini-2.5-flash", None, True),+ ("xai/grok-4.6", None, False),+ ("claude-haiku-4-5@20251001", None, False),+ ("openai/gpt-oss-120b-maas", None, False),+ ("gemma/gemma-3-12b-it", None, False),+ ("agent_engine/1234567890", None, False),+ ("openai/tuned-endpoint", {"base_model": "gemini-2.5-flash"}, True),+ ],+)+def test_vertex_generate_content_config_follows_model_route(+ model: str, litellm_params: dict[str, str] | None, expects_native_config: bool+):+ from litellm.llms.vertex_ai.google_genai.transformation import VertexAIGoogleGenAIConfig++ config: Final = ProviderConfigManager.get_provider_google_genai_generate_content_config(+ model=model, provider=litellm.LlmProviders.VERTEX_AI, litellm_params=litellm_params+ )+ if expects_native_config:+ assert isinstance(config, VertexAIGoogleGenAIConfig)+ else:+ assert config is None++ @pytest.mark.parametrize( ("max_parallel_requests", "rpm", "tpm", "default_max_parallel_requests", "expected"), [