Backtestslitellmrun-20261019-033420#41435
[Bug]: Fix "vertex_ai/xai/grok-4.6" entry in "model_prices_and_context_window.json" — missing supports_prompt_caching
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both diffs touch only the pre-existing test file; the cost-map JSON edits (the actual fix to model_prices_and_context_window.json and the backup) were evidently stripped as generated files from both, so the comparison rests on the test changes. Patch A mostly reshuffles the existing tests: it swaps the supports_prompt_caching() check for a get_model_info() lookup (making the first test redundant with the second), adds Final annotations on locals, and adds a couple of cache_read_input_token_cost assertions that were already implied by the filter. It adds no coverage of the backup file, which the issue explicitly calls out. Patch B leaves the existing tests intact and adds a parametrized test that reads both model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json directly from disk and asserts every vertex_ai/xai/grok-* row with a cache-read price sets supports_prompt_caching: true. That test would fail before the fix and pass after, and it is the only one that guards the backup-file requirement and prevents the two files from drifting. Path resolution (parents[2] from tests/test_litellm/) is correct. Using pydantic TypeAdapter to parse JSON is slightly unconventional but harmless. B adds real, issue-targeted test value; A is largely churn.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
11m 47s
Wall-clock time in the container
Full test suite
No new failures
4 passed · 0 skipped · 4 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
27vs 20 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
This entry has cache_read_input_token_cost (5e-07) and cache_read_input_token_cost_above_200k_tokens (1e-06) but no supports_prompt_caching, so supports_prompt_caching() returns false and GET /v1/model/info reports "supports_prompt_caching": false for a Vertex Grok 4.6 deployment.
The same model on the other providers does set it:
xai/grok-4.6→supports_prompt_caching: true(L49979)azure_ai/grok-4.6→supports_prompt_caching: true
xAI prompt caching is provider-managed (automatic on a stable prefix), and the Vertex route does return usage.prompt_tokens_details.cached_tokens > 0 on follow-up turns, which is what the cache-read rate on this entry bills. Callers that gate on /model/info before reusing a prefix therefore skip caching on the Vertex route only.
The whole vertex_ai/xai/grok-* family looks the same — each has a cache-read price and none sets the flag:
| key | cache_read_input_token_cost | supports_prompt_caching |
|---|---|---|
vertex_ai/xai/grok-4.1-fast-non-reasoning | 5e-08 | absent |
vertex_ai/xai/grok-4.1-fast-reasoning | 5e-08 | absent |
vertex_ai/xai/grok-4.20-non-reasoning | 2e-07 | absent |
vertex_ai/xai/grok-4.20-reasoning | 2e-07 | absent |
vertex_ai/xai/grok-4.3 | 2e-07 | absent |
vertex_ai/xai/grok-4.6 | 5e-07 | absent |
We also need to update model_prices_and_context_window_backup.json.
Confirmed on v1.102.0-rc.1 and current main.
User Flow
Before a (hypothetical) fix
The flow fails when I ask the gateway whether Vertex Grok 4.6 supports prompt caching: the catalog says no, even though a follow-up completion on that same model returns cached tokens.
- I start the proxy with a config that exposes
grok-4.6-vertex(vertex_ai/xai/grok-4.6) andgrok-4.6-direct(xai/grok-4.6) and see it listening onhttp://localhost:4000. - I send
GET http://localhost:4000/v1/modelswithAuthorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names indata. - I send
GET http://localhost:4000/v1/model/infowith the same header. I get HTTP 200. Forgrok-4.6-vertex,model_info.supports_prompt_cachingisfalse(or absent) whilecache_read_input_token_costis5e-07. Forgrok-4.6-direct,supports_prompt_cachingistrue. - I send
POST http://localhost:4000/v1/chat/completionsforgrok-4.6-vertexwith a long stable system prefix, a first user turn, and headerx-grok-conv-id: conv_repro_1. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensof0. - I send the same URL again with the same
x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensgreater than0. - I send
POST http://localhost:4000/v1/responsesforgrok-4.6-vertexwith the same prefix and"prompt_cache_key": "conv_repro_1". I get HTTP 200 andusage.input_tokens_details.cached_tokensof0. - I send the same URL again with the same
prompt_cache_keyand the follow-up turn. I get HTTP 200 andusage.input_tokens_details.cached_tokensgreater than0.
After a (hypothetical) fix
The flow succeeds at the catalog check: Vertex Grok 4.6 is listed as supporting prompt caching, matching the live cache hits on the next turns.
- I start the proxy with a config that exposes
grok-4.6-vertex(vertex_ai/xai/grok-4.6) andgrok-4.6-direct(xai/grok-4.6) and see it listening onhttp://localhost:4000. - I send
GET http://localhost:4000/v1/modelswithAuthorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names indata. - I send
GET http://localhost:4000/v1/model/infowith the same header. I get HTTP 200. For bothgrok-4.6-vertexandgrok-4.6-direct,model_info.supports_prompt_cachingistrue, and the Vertex row still showscache_read_input_token_costof5e-07. - I send
POST http://localhost:4000/v1/chat/completionsforgrok-4.6-vertexwith a long stable system prefix, a first user turn, and headerx-grok-conv-id: conv_repro_1. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensof0. - I send the same URL again with the same
x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 andusage.prompt_tokens_details.cached_tokensgreater than0. - I send
POST http://localhost:4000/v1/responsesforgrok-4.6-vertexwith the same prefix and"prompt_cache_key": "conv_repro_1". I get HTTP 200 andusage.input_tokens_details.cached_tokensof0. - I send the same URL again with the same
prompt_cache_keyand the follow-up turn. I get HTTP 200 andusage.input_tokens_details.cached_tokensgreater than0.
Proof the bug occurs
# config.yaml
model_list:
- model_name: grok-4.6-vertex
litellm_params:
model: vertex_ai/xai/grok-4.6
vertex_project: YOUR_GCP_PROJECT
vertex_location: global
vertex_credentials: /path/to/sa.json
- model_name: grok-4.6-direct
litellm_params:
model: xai/grok-4.6
api_key: os.environ/XAI_API_KEY
# version captured: litellm==1.102.0rc1
# (also reproduced on GitHub main cost map and PyPI 1.103.0.dev1)
export VERTEXAI_PROJECT=YOUR_GCP_PROJECT
# VERTEX credentials via vertex_credentials file above; do not paste the JSON
litellm --config config.yaml --port 4000
# Catalog (no LLM $): Vertex row denies caching, direct xAI row advertises it
curl -sS http://localhost:4000/model/info \
-H "Authorization: Bearer sk-..." | jq '.data[] | select(.model_name|test("grok-4.6")) | {model_name, model: .litellm_params.model, supports_prompt_caching: .model_info.supports_prompt_caching, cache_read_input_token_cost: .model_info.cache_read_input_token_cost}'
{
"model_name": "grok-4.6-vertex",
"model": "vertex_ai/xai/grok-4.6",
"supports_prompt_caching": false,
"cache_read_input_token_cost": 5e-07
}
{
"model_name": "grok-4.6-direct",
"model": "xai/grok-4.6",
"supports_prompt_caching": true,
"cache_read_input_token_cost": 5e-07
}
# Chat Completions
curl -sS http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-H "x-grok-conv-id: conv_repro_1" \
-d '{"model":"grok-4.6-vertex","messages":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}]}'
# second turn: append assistant + follow-up, same header
# Responses
curl -sS http://localhost:4000/v1/responses \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{"model":"grok-4.6-vertex","input":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}],"prompt_cache_key":"conv_repro_1"}'
On the follow-up, the live usage block looks like (numbers from a real Vertex hit; paste your own):
- Chat: "prompt_tokens_details": {"cached_tokens": <nonzero>}
- Responses: "input_tokens_details": {"cached_tokens": <nonzero>}
That is the mismatch: usage says cache hit, /model/info says supports_prompt_caching: false.
What part of LiteLLM is this about?
- Proxy catalog / GET /model/info
- Vertex AI xAI (Grok) partner models
- Prompt caching metadata vs live cached_tokens
What part of LiteLLM is this about?
SDK (litellm Python package)
What LiteLLM version are you on ?
v1.102.0rc1
Twitter / LinkedIn details
_No response_
The two patches
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0..2daae85574 100644@@ -1,6 +1,8 @@+from pathlib import Path from typing import Final import pytest+from pydantic import TypeAdapter import litellm from litellm import get_model_info@@ -9,6 +11,8 @@ from litellm.utils import supports_prompt_caching MODEL: Final = "vertex_ai/xai/grok-4.6" GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-")+REPO_ROOT: Final = Path(__file__).parents[2]+COST_MAP_ADAPTER: Final = TypeAdapter(dict[str, dict[str, object]]) @pytest.mark.usefixtures("local_model_cost_map")@@ -36,3 +40,26 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None assert info.get("supports_prompt_caching") is True assert supports_prompt_caching(model=MODEL) is True+++@pytest.mark.parametrize(+ "cost_map_path",+ (+ REPO_ROOT / "model_prices_and_context_window.json",+ REPO_ROOT / "litellm" / "model_prices_and_context_window_backup.json",+ ),+ ids=("source", "backup"),+)+def test_vertex_ai_grok_rows_flag_prompt_caching_in_both_cost_maps(cost_map_path: Path) -> None:+ cost_map = COST_MAP_ADAPTER.validate_json(cost_map_path.read_bytes())+ cached_vertex_grok_rows = {+ key: entry+ for key, entry in cost_map.items()+ if key.startswith("vertex_ai/xai/grok-") and entry.get("cache_read_input_token_cost")+ }+ assert MODEL in cached_vertex_grok_rows++ missing_flag = tuple(+ key for key, entry in cached_vertex_grok_rows.items() if entry.get("supports_prompt_caching") is not True+ )+ assert missing_flag == ()
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0c..2d5936fabdb 100644@@ -13,16 +13,18 @@ GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-" @pytest.mark.usefixtures("local_model_cost_map") def test_grok_models_with_cache_read_price_advertise_prompt_caching() -> None:- cached_grok_models = tuple(+ cached_grok_models: Final = tuple( key for key, entry in litellm.model_cost.items() if key.startswith(GROK_KEY_PREFIXES) and entry.get("cache_read_input_token_cost") ) assert cached_grok_models, "expected at least one grok model with a cache read price" - missing_flag = tuple(key for key in cached_grok_models if supports_prompt_caching(model=key) is not True)+ missing_flag: Final = tuple(+ key for key in cached_grok_models if get_model_info(model=key).get("supports_prompt_caching") is not True+ ) assert missing_flag == (), (- f"grok models with cache_read_input_token_cost fail supports_prompt_caching: {missing_flag}"+ f"grok models with cache_read_input_token_cost fail get_model_info supports_prompt_caching: {missing_flag}" ) @@ -31,8 +33,14 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None routed_model, provider, _, _ = get_llm_provider(model=MODEL) assert (routed_model, provider) == ("xai/grok-4.6", "vertex_ai") - info = get_model_info(model=routed_model, custom_llm_provider=provider)- assert info["litellm_provider"] == "vertex_ai"- assert info.get("supports_prompt_caching") is True+ routed_info: Final = get_model_info(model=routed_model, custom_llm_provider=provider)+ assert routed_info["litellm_provider"] == "vertex_ai"+ assert routed_info.get("supports_prompt_caching") is True+ assert routed_info.get("cache_read_input_token_cost")++ catalog_info: Final = get_model_info(model=MODEL)+ assert catalog_info["key"] == MODEL+ assert catalog_info.get("supports_prompt_caching") is True+ assert catalog_info.get("cache_read_input_token_cost") assert supports_prompt_caching(model=MODEL) is True