Backtestslitellmrun-20261019-033420#41435

Issue #41435 Merged fix PR #41526 Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: Fix "vertex_ai/xai/grok-4.6" entry in "model_prices_and_context_window.json" — missing supports_prompt_caching

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both diffs touch only the pre-existing test file; the cost-map JSON edits (the actual fix to model_prices_and_context_window.json and the backup) were evidently stripped as generated files from both, so the comparison rests on the test changes. Patch A mostly reshuffles the existing tests: it swaps the supports_prompt_caching() check for a get_model_info() lookup (making the first test redundant with the second), adds Final annotations on locals, and adds a couple of cache_read_input_token_cost assertions that were already implied by the filter. It adds no coverage of the backup file, which the issue explicitly calls out. Patch B leaves the existing tests intact and adds a parametrized test that reads both model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json directly from disk and asserts every vertex_ai/xai/grok-* row with a cache-read price sets supports_prompt_caching: true. That test would fail before the fix and pass after, and it is the only one that guards the backup-file requirement and prevents the two files from drifting. Path resolution (parents[2] from tests/test_litellm/) is correct. Using pydantic TypeAdapter to parse JSON is slightly unconventional but harmless. B adds real, issue-targeted test value; A is largely churn.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

11m 47s

Wall-clock time in the container

Full test suite

No new failures

4 passed · 0 skipped · 4 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

27vs 20 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

https://github.com/BerriAI/litellm/blob/v1.102.0-rc.1/model_prices_and_context_window.json#L48398-L48416

This entry has cache_read_input_token_cost (5e-07) and cache_read_input_token_cost_above_200k_tokens (1e-06) but no supports_prompt_caching, so supports_prompt_caching() returns false and GET /v1/model/info reports "supports_prompt_caching": false for a Vertex Grok 4.6 deployment.

The same model on the other providers does set it:

  • xai/grok-4.6 → supports_prompt_caching: true (L49979)
  • azure_ai/grok-4.6 → supports_prompt_caching: true

xAI prompt caching is provider-managed (automatic on a stable prefix), and the Vertex route does return usage.prompt_tokens_details.cached_tokens > 0 on follow-up turns, which is what the cache-read rate on this entry bills. Callers that gate on /model/info before reusing a prefix therefore skip caching on the Vertex route only.

The whole vertex_ai/xai/grok-* family looks the same — each has a cache-read price and none sets the flag:

keycache_read_input_token_costsupports_prompt_caching
vertex_ai/xai/grok-4.1-fast-non-reasoning5e-08absent
vertex_ai/xai/grok-4.1-fast-reasoning5e-08absent
vertex_ai/xai/grok-4.20-non-reasoning2e-07absent
vertex_ai/xai/grok-4.20-reasoning2e-07absent
vertex_ai/xai/grok-4.32e-07absent
vertex_ai/xai/grok-4.65e-07absent

We also need to update model_prices_and_context_window_backup.json.

Confirmed on v1.102.0-rc.1 and current main.

User Flow

Before a (hypothetical) fix

The flow fails when I ask the gateway whether Vertex Grok 4.6 supports prompt caching: the catalog says no, even though a follow-up completion on that same model returns cached tokens.

  1. I start the proxy with a config that exposes grok-4.6-vertex (vertex_ai/xai/grok-4.6) and grok-4.6-direct (xai/grok-4.6) and see it listening on http://localhost:4000.
  2. I send GET http://localhost:4000/v1/models with Authorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names in data.
  3. I send GET http://localhost:4000/v1/model/info with the same header. I get HTTP 200. For grok-4.6-vertex, model_info.supports_prompt_caching is false (or absent) while cache_read_input_token_cost is 5e-07. For grok-4.6-direct, supports_prompt_caching is true.
  4. I send POST http://localhost:4000/v1/chat/completions for grok-4.6-vertex with a long stable system prefix, a first user turn, and header x-grok-conv-id: conv_repro_1. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens of 0.
  5. I send the same URL again with the same x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens greater than 0.
  6. I send POST http://localhost:4000/v1/responses for grok-4.6-vertex with the same prefix and "prompt_cache_key": "conv_repro_1". I get HTTP 200 and usage.input_tokens_details.cached_tokens of 0.
  7. I send the same URL again with the same prompt_cache_key and the follow-up turn. I get HTTP 200 and usage.input_tokens_details.cached_tokens greater than 0.

After a (hypothetical) fix

The flow succeeds at the catalog check: Vertex Grok 4.6 is listed as supporting prompt caching, matching the live cache hits on the next turns.

  1. I start the proxy with a config that exposes grok-4.6-vertex (vertex_ai/xai/grok-4.6) and grok-4.6-direct (xai/grok-4.6) and see it listening on http://localhost:4000.
  2. I send GET http://localhost:4000/v1/models with Authorization: Bearer sk-[REDACTED]. I get HTTP 200 and both model names in data.
  3. I send GET http://localhost:4000/v1/model/info with the same header. I get HTTP 200. For both grok-4.6-vertex and grok-4.6-direct, model_info.supports_prompt_caching is true, and the Vertex row still shows cache_read_input_token_cost of 5e-07.
  4. I send POST http://localhost:4000/v1/chat/completions for grok-4.6-vertex with a long stable system prefix, a first user turn, and header x-grok-conv-id: conv_repro_1. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens of 0.
  5. I send the same URL again with the same x-grok-conv-id, appending the assistant reply and a follow-up user turn. I get HTTP 200 and usage.prompt_tokens_details.cached_tokens greater than 0.
  6. I send POST http://localhost:4000/v1/responses for grok-4.6-vertex with the same prefix and "prompt_cache_key": "conv_repro_1". I get HTTP 200 and usage.input_tokens_details.cached_tokens of 0.
  7. I send the same URL again with the same prompt_cache_key and the follow-up turn. I get HTTP 200 and usage.input_tokens_details.cached_tokens greater than 0.

Proof the bug occurs

# config.yaml
model_list:
  - model_name: grok-4.6-vertex
    litellm_params:
      model: vertex_ai/xai/grok-4.6
      vertex_project: YOUR_GCP_PROJECT
      vertex_location: global
      vertex_credentials: /path/to/sa.json
  - model_name: grok-4.6-direct
    litellm_params:
      model: xai/grok-4.6
      api_key: os.environ/XAI_API_KEY
# version captured: litellm==1.102.0rc1
# (also reproduced on GitHub main cost map and PyPI 1.103.0.dev1)
export VERTEXAI_PROJECT=YOUR_GCP_PROJECT
# VERTEX credentials via vertex_credentials file above; do not paste the JSON
litellm --config config.yaml --port 4000
# Catalog (no LLM $): Vertex row denies caching, direct xAI row advertises it
curl -sS http://localhost:4000/model/info \
  -H "Authorization: Bearer sk-..." | jq '.data[] | select(.model_name|test("grok-4.6")) | {model_name, model: .litellm_params.model, supports_prompt_caching: .model_info.supports_prompt_caching, cache_read_input_token_cost: .model_info.cache_read_input_token_cost}'
{
  "model_name": "grok-4.6-vertex",
  "model": "vertex_ai/xai/grok-4.6",
  "supports_prompt_caching": false,
  "cache_read_input_token_cost": 5e-07
}
{
  "model_name": "grok-4.6-direct",
  "model": "xai/grok-4.6",
  "supports_prompt_caching": true,
  "cache_read_input_token_cost": 5e-07
}
# Chat Completions
curl -sS http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -H "x-grok-conv-id: conv_repro_1" \
  -d '{"model":"grok-4.6-vertex","messages":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}]}'

# second turn: append assistant + follow-up, same header
# Responses
curl -sS http://localhost:4000/v1/responses \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"grok-4.6-vertex","input":[{"role":"system","content":"<LONG STABLE PREFIX>"},{"role":"user","content":"What is prompt caching?"}],"prompt_cache_key":"conv_repro_1"}'

On the follow-up, the live usage block looks like (numbers from a real Vertex hit; paste your own):

  • Chat: "prompt_tokens_details": {"cached_tokens": <nonzero>}
  • Responses: "input_tokens_details": {"cached_tokens": <nonzero>}

That is the mismatch: usage says cache hit, /model/info says supports_prompt_caching: false.

What part of LiteLLM is this about?

  • Proxy catalog / GET /model/info
  • Vertex AI xAI (Grok) partner models
  • Prompt caching metadata vs live cached_tokens

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.102.0rc1

Twitter / LinkedIn details

_No response_

The two patches

Our patch+27 −0
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0..2daae85574 100644@@ -1,6 +1,8 @@+from pathlib import Path from typing import Final  import pytest+from pydantic import TypeAdapter  import litellm from litellm import get_model_info@@ -9,6 +11,8 @@ from litellm.utils import supports_prompt_caching  MODEL: Final = "vertex_ai/xai/grok-4.6" GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-")+REPO_ROOT: Final = Path(__file__).parents[2]+COST_MAP_ADAPTER: Final = TypeAdapter(dict[str, dict[str, object]])   @pytest.mark.usefixtures("local_model_cost_map")@@ -36,3 +40,26 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None     assert info.get("supports_prompt_caching") is True      assert supports_prompt_caching(model=MODEL) is True+++@pytest.mark.parametrize(+    "cost_map_path",+    (+        REPO_ROOT / "model_prices_and_context_window.json",+        REPO_ROOT / "litellm" / "model_prices_and_context_window_backup.json",+    ),+    ids=("source", "backup"),+)+def test_vertex_ai_grok_rows_flag_prompt_caching_in_both_cost_maps(cost_map_path: Path) -> None:+    cost_map = COST_MAP_ADAPTER.validate_json(cost_map_path.read_bytes())+    cached_vertex_grok_rows = {+        key: entry+        for key, entry in cost_map.items()+        if key.startswith("vertex_ai/xai/grok-") and entry.get("cache_read_input_token_cost")+    }+    assert MODEL in cached_vertex_grok_rows++    missing_flag = tuple(+        key for key, entry in cached_vertex_grok_rows.items() if entry.get("supports_prompt_caching") is not True+    )+    assert missing_flag == () 
Merged fix · PR #41526+14 −6
tests/test_litellm/test_vertex_ai_xai_grok_prompt_caching_metadata.pyindex 72e98711f0c..2d5936fabdb 100644@@ -13,16 +13,18 @@ GROK_KEY_PREFIXES: Final = ("vertex_ai/xai/grok-", "azure_ai/grok-", "xai/grok-"  @pytest.mark.usefixtures("local_model_cost_map") def test_grok_models_with_cache_read_price_advertise_prompt_caching() -> None:-    cached_grok_models = tuple(+    cached_grok_models: Final = tuple(         key         for key, entry in litellm.model_cost.items()         if key.startswith(GROK_KEY_PREFIXES) and entry.get("cache_read_input_token_cost")     )     assert cached_grok_models, "expected at least one grok model with a cache read price" -    missing_flag = tuple(key for key in cached_grok_models if supports_prompt_caching(model=key) is not True)+    missing_flag: Final = tuple(+        key for key in cached_grok_models if get_model_info(model=key).get("supports_prompt_caching") is not True+    )     assert missing_flag == (), (-        f"grok models with cache_read_input_token_cost fail supports_prompt_caching: {missing_flag}"+        f"grok models with cache_read_input_token_cost fail get_model_info supports_prompt_caching: {missing_flag}"     )  @@ -31,8 +33,14 @@ def test_vertex_ai_grok_4_6_supports_prompt_caching_via_get_model_info() -> None     routed_model, provider, _, _ = get_llm_provider(model=MODEL)     assert (routed_model, provider) == ("xai/grok-4.6", "vertex_ai") -    info = get_model_info(model=routed_model, custom_llm_provider=provider)-    assert info["litellm_provider"] == "vertex_ai"-    assert info.get("supports_prompt_caching") is True+    routed_info: Final = get_model_info(model=routed_model, custom_llm_provider=provider)+    assert routed_info["litellm_provider"] == "vertex_ai"+    assert routed_info.get("supports_prompt_caching") is True+    assert routed_info.get("cache_read_input_token_cost")++    catalog_info: Final = get_model_info(model=MODEL)+    assert catalog_info["key"] == MODEL+    assert catalog_info.get("supports_prompt_caching") is True+    assert catalog_info.get("cache_read_input_token_cost")      assert supports_prompt_caching(model=MODEL) is True