Backtestslitellmrun-20261111-220245#48743

Issue #48743 Merged fix PR #48745 Run 11 Nov 2026 · run-20261111-220245 BerriAI/litellm

[Bug]: Gemini grounding annotations drop additional sources for a passage

Tie

The judges split evenly between our patch and the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches fix the root cause the same way: they replace the `chunk_indices[0]` lookup with a loop over every index in `groundingChunkIndices`, skipping indices absent from `chunk_to_uri_map`. The independent tests confirm both pass all three scenarios (two web sources, non-web first source, several multi-source passages) where base fails. Correctness is equal; Patch B dropping the `chunk_indices` truthiness check is harmless since iterating an empty list emits nothing.

The difference is in tests and craft. Patch B adds two explicit, separately named tests that map one-to-one onto the two scenarios called out in the issue (multiple web sources per passage, and web sources following a non-web first index), with literal expected annotations that are easy to read and that reproduce the issue's own example exactly. They are placed next to the existing `test_vertex_ai_annotation_conversion` tests, matching the surrounding file's conventions. Patch A's single parametrized test does cover both scenarios (via `[2, 1, 0]` with a retrievedContext chunk) plus an extra support, but its expected value is built by a list comprehension over the input, which is harder to read and edges toward re-deriving the result rather than stating it. Patch B's early `continue` on a missing segment also flattens nesting and its comment states why multiple annotations are emitted.

These are small margins, but B's tests communicate the fix more directly and sit with related tests, so B is the better patch.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

5m 44s

Wall-clock time in the container

Full test suite

Not run

The harness did not reach the test stage

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

63vs 39 merged

Changed lines, ours vs merged

Description

Gemini can attach several sources to the same grounded passage, but LiteLLM converts only the first source index to an annotation. The other source associations are lost even when Google returned them

Config

This reproduces directly in the Google grounding converter, without credentials or a proxy configuration

LiteLLM Version

1.109.0, current main at f1795e6d447a4e5a3660de0d2fde5e1f44f52929

Steps to Repro

Run the following against an unmodified checkout

from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig

metadata = {
    "groundingChunks": [
        {"web": {"uri": "https://example.com/a", "title": "A"}},
        {"web": {"uri": "https://example.com/b", "title": "B"}},
    ],
    "groundingSupports": [
        {
            "segment": {"startIndex": 0, "endIndex": 5},
            "groundingChunkIndices": [0, 1],
        }
    ],
}
annotations = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "Claim")
print(annotations)

Actual output

[{'type': 'url_citation', 'url_citation': {'start_index': 0, 'end_index': 5, 'url': 'https://example.com/a', 'title': 'A'}}]

Expected output

[
    {'type': 'url_citation', 'url_citation': {'start_index': 0, 'end_index': 5, 'url': 'https://example.com/a', 'title': 'A'}},
    {'type': 'url_citation', 'url_citation': {'start_index': 0, 'end_index': 5, 'url': 'https://example.com/b', 'title': 'B'}},
]

The converter reads chunk_indices[0] instead of iterating over all source indices. If the first index points to a non-web chunk, later valid web sources are lost too

This reproduces locally using synthetic grounding metadata. Fresh direct Gemini API calls through a local LiteLLM proxy also reproduced it on 2026-10-28. The unmodified base retained 12 of 18 source associations. The fixed converter retained all 19 of 19 source associations in a separate generation, with matching URLs, titles and text ranges. Each call contained six passages with multiple web sources, and counts were compared within each response

Which part of LiteLLM is this about?

LLM translation: a specific provider's request or response

How are you deploying?

pip / Python SDK

Related streaming issue #48744 covers citation events after Chat Completions conversion

The two patches

Our patch+58 −5
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.pyindex 9acbc9c459..1443f3bf98 100644@@ -2742,11 +2742,14 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):                 # Get the chunk indices for this support                 chunk_indices = support.get("groundingChunkIndices", []) -                if start_index is not None and end_index is not None and chunk_indices:-                    # Use the first chunk's URL for the annotation-                    first_chunk_idx = chunk_indices[0]-                    if first_chunk_idx in chunk_to_uri_map:-                        uri_info = chunk_to_uri_map[first_chunk_idx]+                if start_index is None or end_index is None:+                    continue++                # A passage can be backed by several sources; emit one+                # annotation per web source so none are dropped+                for chunk_idx in chunk_indices:+                    if chunk_idx in chunk_to_uri_map:+                        uri_info = chunk_to_uri_map[chunk_idx]                          url_citation: ChatCompletionAnnotationURLCitation = {                             "start_index": start_index,tests/unit/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.pyindex 065d894e65..0aeefdc901 100644@@ -3514,6 +3514,56 @@ def test_vertex_ai_annotation_conversion():     assert "Current time" in time_annotation["url_citation"]["title"]  +def test_vertex_ai_annotation_keeps_every_source_of_a_passage():+    from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (+        VertexGeminiConfig,+    )++    metadata = {+        "groundingChunks": [+            {"web": {"uri": "https://example.com/a", "title": "A"}},+            {"web": {"uri": "https://example.com/b", "title": "B"}},+        ],+        "groundingSupports": [{"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": [0, 1]}],+    }++    annotations = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "Claim")++    assert annotations == [+        {+            "type": "url_citation",+            "url_citation": {"start_index": 0, "end_index": 5, "url": "https://example.com/a", "title": "A"},+        },+        {+            "type": "url_citation",+            "url_citation": {"start_index": 0, "end_index": 5, "url": "https://example.com/b", "title": "B"},+        },+    ], annotations+++def test_vertex_ai_annotation_keeps_web_sources_after_a_non_web_first_source():+    from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (+        VertexGeminiConfig,+    )++    metadata = {+        "groundingChunks": [+            {"retrievedContext": {"uri": "gs://bucket/doc.pdf", "title": "Doc"}},+            {"web": {"uri": "https://example.com/b", "title": "B"}},+        ],+        "groundingSupports": [{"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": [0, 1]}],+    }++    annotations = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "Claim")++    assert annotations == [+        {+            "type": "url_citation",+            "url_citation": {"start_index": 0, "end_index": 5, "url": "https://example.com/b", "title": "B"},+        },+    ], annotations++ def test_vertex_ai_annotation_empty_grounding_metadata():     """     Test handling of empty or missing grounding metadata. 
Merged fix · PR #48745+35 −4
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.pyindex 9acbc9c459c..f27593dc94e 100644@@ -2743,10 +2743,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):                 chunk_indices = support.get("groundingChunkIndices", [])                  if start_index is not None and end_index is not None and chunk_indices:-                    # Use the first chunk's URL for the annotation-                    first_chunk_idx = chunk_indices[0]-                    if first_chunk_idx in chunk_to_uri_map:-                        uri_info = chunk_to_uri_map[first_chunk_idx]+                    for chunk_index in chunk_indices:+                        if chunk_index not in chunk_to_uri_map:+                            continue+                        uri_info = chunk_to_uri_map[chunk_index]                          url_citation: ChatCompletionAnnotationURLCitation = {                             "start_index": start_index,tests/unit/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.pyindex 065d894e658..1546d1e9fe0 100644@@ -10655,3 +10655,34 @@ async def test_gemini_sends_a_boolean_for_a_gateway_default_written_as_text_or_n     sent: Final = await _sent_thinking_config("gemini-3.8-flash", {"reasoning_effort": "high"})      assert sent["includeThoughts"] is expected+++@pytest.mark.parametrize("chunk_indices", [[0, 1], [2, 1, 0]])+def test_grounding_support_preserves_every_web_source(chunk_indices: list[int]) -> None:+    metadata: Final = {+        "groundingChunks": [+            {"web": {"uri": "https://example.com/a", "title": "A"}},+            {"web": {"uri": "https://example.com/b", "title": "B"}},+            {"retrievedContext": {"uri": "gs://document"}},+        ],+        "groundingSupports": [+            {"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": chunk_indices},+            {"segment": {"startIndex": 6, "endIndex": 11}, "groundingChunkIndices": [1]},+        ],+    }++    annotations: Final = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "First claim")++    expected_sources: Final = [(index, 0, 5) for index in chunk_indices if index in (0, 1)] + [(1, 6, 11)]+    assert annotations == [+        {+            "type": "url_citation",+            "url_citation": {+                "start_index": start,+                "end_index": end,+                "url": metadata["groundingChunks"][index]["web"]["uri"],+                "title": metadata["groundingChunks"][index]["web"]["title"],+            },+        }+        for index, start, end in expected_sources+    ]