Backtestslitellmrun-20261111-220245#41435

Issue #41435 Merged fix PR #41526 Run 11 Nov 2026 · run-20261111-220245 BerriAI/litellm

[Bug]: Fix "vertex_ai/xai/grok-4.6" entry in "model_prices_and_context_window.json" — missing supports_prompt_caching

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both diffs contain only test changes; the JSON cost-map edits were evidently stripped as data files, so the comparison rests on the tests. Patch A rewrites the existing test to go through get_model_info instead of supports_prompt_caching and adds a few redundant assertions on the routed/catalog info for grok-4.6, but it removes nothing of substance and adds no coverage of the backup file the issue explicitly asks to be updated. Patch B leaves the existing tests intact and adds a parametrized test that reads both model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json directly from disk and asserts all six vertex_ai/xai/grok-* rows named in the issue carry both a cache-read price and supports_prompt_caching: true. That test fails before the fix and passes after, pins the exact model list from the issue, and catches a source/backup drift that A's model_cost-based checks would miss. REPO_ROOT resolution (parents[2] from tests/test_litellm/) is correct. The pydantic TypeAdapter for parsing is slightly heavier than json.loads but harmless. B's tests more faithfully cover the issue's full ask with no churn to existing tests.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

15m 9s

Wall-clock time in the container

Full test suite

No new failures

4 passed · 0 skipped · 4 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

32vs 20 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

https://github.com/BerriAI/litellm/blob/v1.102.0-rc.1/model_prices_and_context_window.json#L48398-L48416

This entry has cache_read_input_token_cost (5e-07) and cache_read_input_token_cost_above_200k_tokens (1e-06) but no supports_prompt_caching, so supports_prompt_caching() returns false and GET /v1/model/info reports "supports_prompt_caching": false for a Vertex Grok 4.6 deployment.

The same model on the other providers does set it:

  • xai/grok-4.6 → supports_prompt_caching: true (L49979)
  • azure_ai/grok-4.6 → supports_prompt_caching: true

xAI prompt caching is provider-managed (automatic on a stable prefix), and the Vertex route does return usage.prompt_tokens_details.cached_tokens > 0 on follow-up turns, which is what the cache-read rate on this entry bills. Callers that gate on /model/info before reusing a prefix therefore skip caching on the Vertex route only.

The whole vertex_ai/xai/grok-* family looks the same — each has a cache-read price and none sets the flag:

keycache_read_input_token_costsupports_prompt_caching
vertex_ai/xai/grok-4.1-fast-non-reasoning5e-08absent
vertex_ai/xai/grok-4.1-fast-reasoning5e-08absent
vertex_ai/xai/grok-4.20-non-reasoning2e-07absent
vertex_ai/xai/grok-4.20-reasoning2e-07absent
vertex_ai/xai/grok-4.32e-07absent
vertex_ai/xai/grok-4.65e-07absent

We also need to update model_prices_and_context_window_backup.json.

Confirmed on v1.102.0-rc.1 and current main.

User Flow

Before a (hypothetical) fix

The flow fails when I ask the gateway whether Vertex Grok 4.6 supports prompt caching: the catalog says no, even though a follow-up completion on that same model returns cached tokens.

  1. I start the proxy with a config that exposes grok-4.6-vertex (vertex_ai/xai/grok-4.6) and grok-4.6-direct (xai/grok-4.6) and see it listening on http://localhost:4000.
  2. I send GET http://localhost:4000/v1/models with Authorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names in data.
  3. I send GET http://localhost:4000/v1/model/info with the same header. I get HTTP 200. For grok-4.6-vertex, model_info.supports_prompt_caching is false (or absent) while cache_read_input_token_cost is 5e-07. For grok-4.6-direct, supports_prompt_caching is true.
  4. I send POST http://localhost:4000/v1/chat/completions for grok-4.6-vertex with a long stable system prefix, a first user turn, and header x-grok-conv-id: conv_repro_1. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens of 0.
  5. I send the same URL again with the same x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens greater than 0.
  6. I send POST http://localhost:4000/v1/responses for grok-4.6-vertex with the same prefix and "prompt_cache_key": "conv_repro_1". I get HTTP 200 and usage.input_tokens_details.cached_tokens of 0.
  7. I send the same URL again with the same prompt_cache_key and the follow-up turn. I get HTTP 200 and usage.input_tokens_details.cached_tokens greater than 0.

After a (hypothetical) fix

The flow succeeds at the catalog check: Vertex Grok 4.6 is listed as supporting prompt caching, matching the live cache hits on the next turns.

  1. I start the proxy with a config that exposes grok-4.6-vertex (vertex_ai/xai/grok-4.6) and grok-4.6-direct (xai/grok-4.6) and see it listening on http://localhost:4000.
  2. I send GET http://localhost:4000/v1/models with Authorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names in data.
  3. I send GET http://localhost:4000/v1/model/info with the same header. I get HTTP 200. For both grok-4.6-vertex and grok-4.6-direct, model_info.supports_prompt_caching is true, and the Vertex row still shows cache_read_input_token_cost of 5e-07.
  4. I send POST http://localhost:4000/v1/chat/completions for grok-4.6-vertex with a long stable system prefix, a first user turn, and header x-grok-conv-id: conv_repro_1. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens of 0.
  5. I send the same URL again with the same x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens greater than 0.
  6. I send POST http://localhost:4000/v1/responses for grok-4.6-vertex with the same prefix and "prompt_cache_key": "conv_repro_1". I get HTTP 200 and usage.input_tokens_details.cached_tokens of 0.
  7. I send the same URL again with the same prompt_cache_key and the follow-up turn. I get HTTP 200 and usage.input_tokens_details.cached_tokens greater than 0.

Proof the bug occurs

# config.yaml
model_list:
  - model_name: grok-4.6-vertex
    litellm_params:
      model: vertex_ai/xai/grok-4.6
      vertex_project: YOUR_GCP_PROJECT
      vertex_location: global
      vertex_credentials: /path/to/sa.json
  - model_name: grok-4.6-direct
    litellm_params:
      model: xai/grok-4.6
      api_key: os.environ/XAI_API_KEY
# version captured: litellm==1.102.0rc1
# (also reproduced on GitHub main cost map and PyPI 1.103.0.dev1)
export VERTEXAI_PROJECT=YOUR_GCP_PROJECT
# VERTEX credentials via vertex_credentials file above; do not paste the JSON
litellm --config config.yaml --port 4000
# Catalog (no LLM $): Vertex row denies caching, direct xAI row advertises it
curl -sS http://localhost:4000/model/info \
  -H "Authorization: Bearer sk-..." | jq '.data[] | select(.model_name|test("grok-4.6")) | {model_name, model: .litellm_params.model, supports_prompt_caching: .model_info.supports_prompt_caching, cache_read_input_token_cost: .model_info.cache_read_input_token_cost}'
{
  "model_name": "grok-4.6-vertex",
  "model": "vertex_ai/xai/grok-4.6",
  "supports_prompt_caching": false,
  "cache_read_input_token_cost": 5e-07
}
{
  "model_name": "grok-4.6-direct",
  "model": "xai/grok-4.6",
  "supports_prompt_caching": true,
  "cache_read_input_token_cost": 5e-07
}
# Chat Completions
curl -sS http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -H "x-grok-conv-id: conv_repro_1" \
  -d '{"model":"grok-4.6-vertex","messages":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}]}'

# second turn: append assistant + follow-up, same header
# Responses
curl -sS http://localhost:4000/v1/responses \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"grok-4.6-vertex","input":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}],"prompt_cache_key":"conv_repro_1"}'

On the follow-up, the live usage block looks like (numbers from a real Vertex hit; paste your own):

  • Chat: "prompt_tokens_details": {"cached_tokens": <nonzero>}
  • Responses: "input_tokens_details": {"cached_tokens": <nonzero>}

That is the mismatch: usage says cache hit, /model/info says supports_prompt_caching: false.

What part of LiteLLM is this about?

  • Proxy catalog / GET /model/info
  • Vertex AI xAI (Grok) partner models
  • Prompt caching metadata vs live cached_tokens

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.102.0rc1

Twitter / LinkedIn details

_No response_

The two patches

Our patch+32 −0
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0..6f80bc7722 100644@@ -1,6 +1,8 @@+from pathlib import Path from typing import Final  import pytest+from pydantic import TypeAdapter  import litellm from litellm import get_model_info@@ -9,6 +11,16 @@ from litellm.utils import supports_prompt_caching  MODEL: Final = "vertex_ai/xai/grok-4.6" GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-")+REPO_ROOT: Final = Path(__file__).parents[2]+COST_MAP_ADAPTER: Final = TypeAdapter(dict[str, dict[str, object]])+VERTEX_GROK_MODELS: Final = (+    "vertex_ai/xai/grok-4.1-fast-non-reasoning",+    "vertex_ai/xai/grok-4.1-fast-reasoning",+    "vertex_ai/xai/grok-4.20-non-reasoning",+    "vertex_ai/xai/grok-4.20-reasoning",+    "vertex_ai/xai/grok-4.3",+    "vertex_ai/xai/grok-4.6",+)   @pytest.mark.usefixtures("local_model_cost_map")@@ -36,3 +48,23 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None     assert info.get("supports_prompt_caching") is True      assert supports_prompt_caching(model=MODEL) is True+++@pytest.mark.parametrize(+    "cost_map_path",+    (+        REPO_ROOT / "model_prices_and_context_window.json",+        REPO_ROOT / "litellm" / "model_prices_and_context_window_backup.json",+    ),+    ids=("source", "backup"),+)+def test_vertex_ai_grok_rows_flag_prompt_caching_in_both_cost_maps(cost_map_path: Path) -> None:+    cost_map = COST_MAP_ADAPTER.validate_json(cost_map_path.read_bytes())++    unflagged = tuple(+        key+        for key in VERTEX_GROK_MODELS+        if not cost_map[key].get("cache_read_input_token_cost")+        or cost_map[key].get("supports_prompt_caching") is not True+    )+    assert unflagged == (), f"{cost_map_path.name}: vertex grok rows missing prompt caching: {unflagged}" 
Merged fix · PR #41526+14 −6
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0c..2d5936fabdb 100644@@ -13,16 +13,18 @@ GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-"  @pytest.mark.usefixtures("local_model_cost_map") def test_grok_models_with_cache_read_price_advertise_prompt_caching() -> None:-    cached_grok_models = tuple(+    cached_grok_models: Final = tuple(         key         for key, entry in litellm.model_cost.items()         if key.startswith(GROK_KEY_PREFIXES) and entry.get("cache_read_input_token_cost")     )     assert cached_grok_models, "expected at least one grok model with a cache read price" -    missing_flag = tuple(key for key in cached_grok_models if supports_prompt_caching(model=key) is not True)+    missing_flag: Final = tuple(+        key for key in cached_grok_models if get_model_info(model=key).get("supports_prompt_caching") is not True+    )     assert missing_flag == (), (-        f"grok models with cache_read_input_token_cost fail supports_prompt_caching: {missing_flag}"+        f"grok models with cache_read_input_token_cost fail get_model_info supports_prompt_caching: {missing_flag}"     )  @@ -31,8 +33,14 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None     routed_model, provider, _, _ = get_llm_provider(model=MODEL)     assert (routed_model, provider) == ("xai/grok-4.6", "vertex_ai") -    info = get_model_info(model=routed_model, custom_llm_provider=provider)-    assert info["litellm_provider"] == "vertex_ai"-    assert info.get("supports_prompt_caching") is True+    routed_info: Final = get_model_info(model=routed_model, custom_llm_provider=provider)+    assert routed_info["litellm_provider"] == "vertex_ai"+    assert routed_info.get("supports_prompt_caching") is True+    assert routed_info.get("cache_read_input_token_cost")++    catalog_info: Final = get_model_info(model=MODEL)+    assert catalog_info["key"] == MODEL+    assert catalog_info.get("supports_prompt_caching") is True+    assert catalog_info.get("cache_read_input_token_cost")      assert supports_prompt_caching(model=MODEL) is True