Backtestslitellmrun-20261111-220245#48743
[Bug]: Gemini grounding annotations drop additional sources for a passage
The judges split evenly between our patch and the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the root cause the same way: they replace the `chunk_indices[0]` lookup with a loop over every index in `groundingChunkIndices`, skipping indices absent from `chunk_to_uri_map`. The independent tests confirm both pass all three scenarios (two web sources, non-web first source, several multi-source passages) where base fails. Correctness is equal; Patch B dropping the `chunk_indices` truthiness check is harmless since iterating an empty list emits nothing. The difference is in tests and craft. Patch B adds two explicit, separately named tests that map one-to-one onto the two scenarios called out in the issue (multiple web sources per passage, and web sources following a non-web first index), with literal expected annotations that are easy to read and that reproduce the issue's own example exactly. They are placed next to the existing `test_vertex_ai_annotation_conversion` tests, matching the surrounding file's conventions. Patch A's single parametrized test does cover both scenarios (via `[2, 1, 0]` with a retrievedContext chunk) plus an extra support, but its expected value is built by a list comprehension over the input, which is harder to read and edges toward re-deriving the result rather than stating it. Patch B's early `continue` on a missing segment also flattens nesting and its comment states why multiple annotations are emitted. These are small margins, but B's tests communicate the fix more directly and sit with related tests, so B is the better patch.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
5m 44s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
63vs 39 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
Gemini can attach several sources to the same grounded passage, but LiteLLM converts only the first source index to an annotation. The other source associations are lost even when Google returned them
Config
This reproduces directly in the Google grounding converter, without credentials or a proxy configuration
LiteLLM Version
1.109.0, current main at f1795e6d447a4e5a3660de0d2fde5e1f44f52929
Steps to Repro
Run the following against an unmodified checkout
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
metadata = {
"groundingChunks": [
{"web": {"uri": "https://example.com/a", "title": "A"}},
{"web": {"uri": "https://example.com/b", "title": "B"}},
],
"groundingSupports": [
{
"segment": {"startIndex": 0, "endIndex": 5},
"groundingChunkIndices": [0, 1],
}
],
}
annotations = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "Claim")
print(annotations)
Actual output
[{'type': 'url_citation', 'url_citation': {'start_index': 0, 'end_index': 5, 'url': 'https://example.com/a', 'title': 'A'}}]
Expected output
[
{'type': 'url_citation', 'url_citation': {'start_index': 0, 'end_index': 5, 'url': 'https://example.com/a', 'title': 'A'}},
{'type': 'url_citation', 'url_citation': {'start_index': 0, 'end_index': 5, 'url': 'https://example.com/b', 'title': 'B'}},
]
The converter reads chunk_indices[0] instead of iterating over all source indices. If the first index points to a non-web chunk, later valid web sources are lost too
This reproduces locally using synthetic grounding metadata. Fresh direct Gemini API calls through a local LiteLLM proxy also reproduced it on 2026-10-28. The unmodified base retained 12 of 18 source associations. The fixed converter retained all 19 of 19 source associations in a separate generation, with matching URLs, titles and text ranges. Each call contained six passages with multiple web sources, and counts were compared within each response
Which part of LiteLLM is this about?
LLM translation: a specific provider's request or response
How are you deploying?
pip / Python SDK
Related streaming issue #48744 covers citation events after Chat Completions conversion
The two patches
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.pyindex 9acbc9c459..1443f3bf98 100644@@ -2742,11 +2742,14 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): # Get the chunk indices for this support chunk_indices = support.get("groundingChunkIndices", []) - if start_index is not None and end_index is not None and chunk_indices:- # Use the first chunk's URL for the annotation- first_chunk_idx = chunk_indices[0]- if first_chunk_idx in chunk_to_uri_map:- uri_info = chunk_to_uri_map[first_chunk_idx]+ if start_index is None or end_index is None:+ continue++ # A passage can be backed by several sources; emit one+ # annotation per web source so none are dropped+ for chunk_idx in chunk_indices:+ if chunk_idx in chunk_to_uri_map:+ uri_info = chunk_to_uri_map[chunk_idx] url_citation: ChatCompletionAnnotationURLCitation = { "start_index": start_index,tests/unit/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.pyindex 065d894e65..0aeefdc901 100644@@ -3514,6 +3514,56 @@ def test_vertex_ai_annotation_conversion(): assert "Current time" in time_annotation["url_citation"]["title"] +def test_vertex_ai_annotation_keeps_every_source_of_a_passage():+ from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (+ VertexGeminiConfig,+ )++ metadata = {+ "groundingChunks": [+ {"web": {"uri": "https://example.com/a", "title": "A"}},+ {"web": {"uri": "https://example.com/b", "title": "B"}},+ ],+ "groundingSupports": [{"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": [0, 1]}],+ }++ annotations = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "Claim")++ assert annotations == [+ {+ "type": "url_citation",+ "url_citation": {"start_index": 0, "end_index": 5, "url": "https://example.com/a", "title": "A"},+ },+ {+ "type": "url_citation",+ "url_citation": {"start_index": 0, "end_index": 5, "url": "https://example.com/b", "title": "B"},+ },+ ], annotations+++def test_vertex_ai_annotation_keeps_web_sources_after_a_non_web_first_source():+ from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (+ VertexGeminiConfig,+ )++ metadata = {+ "groundingChunks": [+ {"retrievedContext": {"uri": "gs://bucket/doc.pdf", "title": "Doc"}},+ {"web": {"uri": "https://example.com/b", "title": "B"}},+ ],+ "groundingSupports": [{"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": [0, 1]}],+ }++ annotations = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "Claim")++ assert annotations == [+ {+ "type": "url_citation",+ "url_citation": {"start_index": 0, "end_index": 5, "url": "https://example.com/b", "title": "B"},+ },+ ], annotations++ def test_vertex_ai_annotation_empty_grounding_metadata(): """ Test handling of empty or missing grounding metadata.
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.pyindex 9acbc9c459c..f27593dc94e 100644@@ -2743,10 +2743,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): chunk_indices = support.get("groundingChunkIndices", []) if start_index is not None and end_index is not None and chunk_indices:- # Use the first chunk's URL for the annotation- first_chunk_idx = chunk_indices[0]- if first_chunk_idx in chunk_to_uri_map:- uri_info = chunk_to_uri_map[first_chunk_idx]+ for chunk_index in chunk_indices:+ if chunk_index not in chunk_to_uri_map:+ continue+ uri_info = chunk_to_uri_map[chunk_index] url_citation: ChatCompletionAnnotationURLCitation = { "start_index": start_index,tests/unit/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.pyindex 065d894e658..1546d1e9fe0 100644@@ -10655,3 +10655,34 @@ async def test_gemini_sends_a_boolean_for_a_gateway_default_written_as_text_or_n sent: Final = await _sent_thinking_config("gemini-3.8-flash", {"reasoning_effort": "high"}) assert sent["includeThoughts"] is expected+++@pytest.mark.parametrize("chunk_indices", [[0, 1], [2, 1, 0]])+def test_grounding_support_preserves_every_web_source(chunk_indices: list[int]) -> None:+ metadata: Final = {+ "groundingChunks": [+ {"web": {"uri": "https://example.com/a", "title": "A"}},+ {"web": {"uri": "https://example.com/b", "title": "B"}},+ {"retrievedContext": {"uri": "gs://document"}},+ ],+ "groundingSupports": [+ {"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": chunk_indices},+ {"segment": {"startIndex": 6, "endIndex": 11}, "groundingChunkIndices": [1]},+ ],+ }++ annotations: Final = VertexGeminiConfig._convert_grounding_metadata_to_annotations([metadata], "First claim")++ expected_sources: Final = [(index, 0, 5) for index in chunk_indices if index in (0, 1)] + [(1, 6, 11)]+ assert annotations == [+ {+ "type": "url_citation",+ "url_citation": {+ "start_index": start,+ "end_index": end,+ "url": metadata["groundingChunks"][index]["web"]["uri"],+ "title": metadata["groundingChunks"][index]["web"]["title"],+ },+ }+ for index, start, end in expected_sources+ ]