Backtestslitellmrun-20261111-220245#41435
[Bug]: Fix "vertex_ai/xai/grok-4.6" entry in "model_prices_and_context_window.json" — missing supports_prompt_caching
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both diffs contain only test changes; the JSON cost-map edits were evidently stripped as data files, so the comparison rests on the tests. Patch A rewrites the existing test to go through get_model_info instead of supports_prompt_caching and adds a few redundant assertions on the routed/catalog info for grok-4.6, but it removes nothing of substance and adds no coverage of the backup file the issue explicitly asks to be updated. Patch B leaves the existing tests intact and adds a parametrized test that reads both model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json directly from disk and asserts all six vertex_ai/xai/grok-* rows named in the issue carry both a cache-read price and supports_prompt_caching: true. That test fails before the fix and passes after, pins the exact model list from the issue, and catches a source/backup drift that A's model_cost-based checks would miss. REPO_ROOT resolution (parents[2] from tests/test_litellm/) is correct. The pydantic TypeAdapter for parsing is slightly heavier than json.loads but harmless. B's tests more faithfully cover the issue's full ask with no churn to existing tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
15m 9s
Wall-clock time in the container
Full test suite
No new failures
4 passed · 0 skipped · 4 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
32vs 20 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
This entry has cache_read_input_token_cost (5e-07) and cache_read_input_token_cost_above_200k_tokens (1e-06) but no supports_prompt_caching, so supports_prompt_caching() returns false and GET /v1/model/info reports "supports_prompt_caching": false for a Vertex Grok 4.6 deployment.
The same model on the other providers does set it:
xai/grok-4.6→supports_prompt_caching: true(L49979)azure_ai/grok-4.6→supports_prompt_caching: true
xAI prompt caching is provider-managed (automatic on a stable prefix), and the Vertex route does return usage.prompt_tokens_details.cached_tokens > 0 on follow-up turns, which is what the cache-read rate on this entry bills. Callers that gate on /model/info before reusing a prefix therefore skip caching on the Vertex route only.
The whole vertex_ai/xai/grok-* family looks the same — each has a cache-read price and none sets the flag:
| key | cache_read_input_token_cost | supports_prompt_caching |
|---|---|---|
vertex_ai/xai/grok-4.1-fast-non-reasoning | 5e-08 | absent |
vertex_ai/xai/grok-4.1-fast-reasoning | 5e-08 | absent |
vertex_ai/xai/grok-4.20-non-reasoning | 2e-07 | absent |
vertex_ai/xai/grok-4.20-reasoning | 2e-07 | absent |
vertex_ai/xai/grok-4.3 | 2e-07 | absent |
vertex_ai/xai/grok-4.6 | 5e-07 | absent |
We also need to update model_prices_and_context_window_backup.json.
Confirmed on v1.102.0-rc.1 and current main.
User Flow
Before a (hypothetical) fix
The flow fails when I ask the gateway whether Vertex Grok 4.6 supports prompt caching: the catalog says no, even though a follow-up completion on that same model returns cached tokens.
- I start the proxy with a config that exposes
grok-4.6-vertex(vertex_ai/xai/grok-4.6) andgrok-4.6-direct(xai/grok-4.6) and see it listening onhttp://localhost:4000. - I send
GET http://localhost:4000/v1/modelswithAuthorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names indata. - I send
GET http://localhost:4000/v1/model/infowith the same header. I get HTTP 200. Forgrok-4.6-vertex,model_info.supports_prompt_cachingisfalse(or absent) whilecache_read_input_token_costis5e-07. Forgrok-4.6-direct,supports_prompt_cachingistrue. - I send
POST http://localhost:4000/v1/chat/completionsforgrok-4.6-vertexwith a long stable system prefix, a first user turn, and headerx-grok-conv-id: conv_repro_1. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensof0. - I send the same URL again with the same
x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensgreater than0. - I send
POST http://localhost:4000/v1/responsesforgrok-4.6-vertexwith the same prefix and"prompt_cache_key": "conv_repro_1". I get HTTP 200 andusage.input_tokens_details.cached_tokensof0. - I send the same URL again with the same
prompt_cache_keyand the follow-up turn. I get HTTP 200 andusage.input_tokens_details.cached_tokensgreater than0.
After a (hypothetical) fix
The flow succeeds at the catalog check: Vertex Grok 4.6 is listed as supporting prompt caching, matching the live cache hits on the next turns.
- I start the proxy with a config that exposes
grok-4.6-vertex(vertex_ai/xai/grok-4.6) andgrok-4.6-direct(xai/grok-4.6) and see it listening onhttp://localhost:4000. - I send
GET http://localhost:4000/v1/modelswithAuthorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names indata. - I send
GET http://localhost:4000/v1/model/infowith the same header. I get HTTP 200. For bothgrok-4.6-vertexandgrok-4.6-direct,model_info.supports_prompt_cachingistrue, and the Vertex row still showscache_read_input_token_costof5e-07. - I send
POST http://localhost:4000/v1/chat/completionsforgrok-4.6-vertexwith a long stable system prefix, a first user turn, and headerx-grok-conv-id: conv_repro_1. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensof0. - I send the same URL again with the same
x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensgreater than0. - I send
POST http://localhost:4000/v1/responsesforgrok-4.6-vertexwith the same prefix and"prompt_cache_key": "conv_repro_1". I get HTTP 200 andusage.input_tokens_details.cached_tokensof0. - I send the same URL again with the same
prompt_cache_keyand the follow-up turn. I get HTTP 200 andusage.input_tokens_details.cached_tokensgreater than0.
Proof the bug occurs
# config.yaml
model_list:
- model_name: grok-4.6-vertex
litellm_params:
model: vertex_ai/xai/grok-4.6
vertex_project: YOUR_GCP_PROJECT
vertex_location: global
vertex_credentials: /path/to/sa.json
- model_name: grok-4.6-direct
litellm_params:
model: xai/grok-4.6
api_key: os.environ/XAI_API_KEY
# version captured: litellm==1.102.0rc1
# (also reproduced on GitHub main cost map and PyPI 1.103.0.dev1)
export VERTEXAI_PROJECT=YOUR_GCP_PROJECT
# VERTEX credentials via vertex_credentials file above; do not paste the JSON
litellm --config config.yaml --port 4000
# Catalog (no LLM $): Vertex row denies caching, direct xAI row advertises it
curl -sS http://localhost:4000/model/info \
-H "Authorization: Bearer sk-..." | jq '.data[] | select(.model_name|test("grok-4.6")) | {model_name, model: .litellm_params.model, supports_prompt_caching: .model_info.supports_prompt_caching, cache_read_input_token_cost: .model_info.cache_read_input_token_cost}'
{
"model_name": "grok-4.6-vertex",
"model": "vertex_ai/xai/grok-4.6",
"supports_prompt_caching": false,
"cache_read_input_token_cost": 5e-07
}
{
"model_name": "grok-4.6-direct",
"model": "xai/grok-4.6",
"supports_prompt_caching": true,
"cache_read_input_token_cost": 5e-07
}
# Chat Completions
curl -sS http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-H "x-grok-conv-id: conv_repro_1" \
-d '{"model":"grok-4.6-vertex","messages":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}]}'
# second turn: append assistant + follow-up, same header
# Responses
curl -sS http://localhost:4000/v1/responses \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"model":"grok-4.6-vertex","input":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}],"prompt_cache_key":"conv_repro_1"}'
On the follow-up, the live usage block looks like (numbers from a real Vertex hit; paste your own):
- Chat: "prompt_tokens_details": {"cached_tokens": <nonzero>}
- Responses: "input_tokens_details": {"cached_tokens": <nonzero>}
That is the mismatch: usage says cache hit, /model/info says supports_prompt_caching: false.
What part of LiteLLM is this about?
- Proxy catalog / GET /model/info
- Vertex AI xAI (Grok) partner models
- Prompt caching metadata vs live cached_tokens
What part of LiteLLM is this about?
SDK (litellm Python package)
What LiteLLM version are you on ?
v1.102.0rc1
Twitter / LinkedIn details
_No response_
The two patches
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0..6f80bc7722 100644@@ -1,6 +1,8 @@+from pathlib import Path from typing import Final import pytest+from pydantic import TypeAdapter import litellm from litellm import get_model_info@@ -9,6 +11,16 @@ from litellm.utils import supports_prompt_caching MODEL: Final = "vertex_ai/xai/grok-4.6" GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-")+REPO_ROOT: Final = Path(__file__).parents[2]+COST_MAP_ADAPTER: Final = TypeAdapter(dict[str, dict[str, object]])+VERTEX_GROK_MODELS: Final = (+ "vertex_ai/xai/grok-4.1-fast-non-reasoning",+ "vertex_ai/xai/grok-4.1-fast-reasoning",+ "vertex_ai/xai/grok-4.20-non-reasoning",+ "vertex_ai/xai/grok-4.20-reasoning",+ "vertex_ai/xai/grok-4.3",+ "vertex_ai/xai/grok-4.6",+) @pytest.mark.usefixtures("local_model_cost_map")@@ -36,3 +48,23 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None assert info.get("supports_prompt_caching") is True assert supports_prompt_caching(model=MODEL) is True+++@pytest.mark.parametrize(+ "cost_map_path",+ (+ REPO_ROOT / "model_prices_and_context_window.json",+ REPO_ROOT / "litellm" / "model_prices_and_context_window_backup.json",+ ),+ ids=("source", "backup"),+)+def test_vertex_ai_grok_rows_flag_prompt_caching_in_both_cost_maps(cost_map_path: Path) -> None:+ cost_map = COST_MAP_ADAPTER.validate_json(cost_map_path.read_bytes())++ unflagged = tuple(+ key+ for key in VERTEX_GROK_MODELS+ if not cost_map[key].get("cache_read_input_token_cost")+ or cost_map[key].get("supports_prompt_caching") is not True+ )+ assert unflagged == (), f"{cost_map_path.name}: vertex grok rows missing prompt caching: {unflagged}"
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0c..2d5936fabdb 100644@@ -13,16 +13,18 @@ GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-" @pytest.mark.usefixtures("local_model_cost_map") def test_grok_models_with_cache_read_price_advertise_prompt_caching() -> None:- cached_grok_models = tuple(+ cached_grok_models: Final = tuple( key for key, entry in litellm.model_cost.items() if key.startswith(GROK_KEY_PREFIXES) and entry.get("cache_read_input_token_cost") ) assert cached_grok_models, "expected at least one grok model with a cache read price" - missing_flag = tuple(key for key in cached_grok_models if supports_prompt_caching(model=key) is not True)+ missing_flag: Final = tuple(+ key for key in cached_grok_models if get_model_info(model=key).get("supports_prompt_caching") is not True+ ) assert missing_flag == (), (- f"grok models with cache_read_input_token_cost fail supports_prompt_caching: {missing_flag}"+ f"grok models with cache_read_input_token_cost fail get_model_info supports_prompt_caching: {missing_flag}" ) @@ -31,8 +33,14 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None routed_model, provider, _, _ = get_llm_provider(model=MODEL) assert (routed_model, provider) == ("xai/grok-4.6", "vertex_ai") - info = get_model_info(model=routed_model, custom_llm_provider=provider)- assert info["litellm_provider"] == "vertex_ai"- assert info.get("supports_prompt_caching") is True+ routed_info: Final = get_model_info(model=routed_model, custom_llm_provider=provider)+ assert routed_info["litellm_provider"] == "vertex_ai"+ assert routed_info.get("supports_prompt_caching") is True+ assert routed_info.get("cache_read_input_token_cost")++ catalog_info: Final = get_model_info(model=MODEL)+ assert catalog_info["key"] == MODEL+ assert catalog_info.get("supports_prompt_caching") is True+ assert catalog_info.get("cache_read_input_token_cost") assert supports_prompt_caching(model=MODEL) is True