Backtestslitellmrun-20261111-220245#47841
[Bug]: spend_logs_metadata is never recorded for /v1beta/interactions
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches apply the same one-line fix the issue verified (adding "interactions" to LITELLM_METADATA_ROUTES), and all five independent tests pass on both, so they are equal on the reported scenarios. Patch A additionally hardens the merge helper the issue names as root-cause step 3: add_missing_spend_metadata_to_litellm_metadata now carries spend_logs_metadata across when `metadata` is demonstrably the proxy's request bucket (it holds user_api_key stamps) and litellm_metadata does not already define the key. That guard preserves the /responses convention that provider body `metadata` is not recorded (covered by an explicit test), keeps a caller-supplied litellm_metadata value authoritative, and extends the fix to other generic-API routes (videos, vector_stores, containers) where the same split happens, which the issue itself flags as the broader alternative and as the cause of related breakage. The change is small, conditional, and documented, so the added risk is low relative to the coverage it gains. On tests, A is stronger: it exercises the real get_logging_payload path to the spend-log row for both header and body litellm_metadata on both interaction paths, plus unit tests for the helper's new carry and non-carry cases. Patch B's tests are sound but narrower (route classification plus a hand-simulated router update feeding get_litellm_metadata_from_kwargs). Both patches' new tests depend on helpers/imports not visible in the diffs, so that uncertainty is shared. B is a clean minimal fix; A is the same fix plus a targeted, well-guarded root-cause repair with better test coverage, so A is the better patch.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
17m 39s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
154vs 48 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
On the proxy, spend_logs_metadata sent with a Gemini Interactions API request (POST /v1beta/interactions, call_type=acreate_interaction) never reaches LiteLLM_SpendLogs: metadata.spend_logs_metadata is null on every row, whichever way it is sent (x-litellm-spend-logs-metadata header, body metadata.spend_logs_metadata or body litellm_metadata.spend_logs_metadata). The same three ways work on /v1/chat/completions against the same deployment. Tags and the session id do arrive. Anyone who attributes spend through spend_logs_metadata loses it on the Interactions route.
Expected: the header and litellm_metadata (as on /v1/responses) land in metadata.spend_logs_metadata.
Seen on v1.106.1 and v1.104.0.
Root cause (traced on v1.106.1 with a debug hook that logged key names at each step):
interactionsis not inLITELLM_METADATA_ROUTES(litellm/proxy/litellm_pre_call_utils.py), so the proxy puts the request metadata (spend_logs_metadata,tags,user_api_key_*) undermetadata.- The router puts the deployment metadata (
model_group,deployment,model_info,api_base, retry fields) underlitellm_metadata, solitellm.interactions.createreceives both dicts. get_logging_payload(litellm/proxy/spend_tracking/spend_tracking_utils.py) reads them throughget_litellm_metadata_from_kwargs(litellm/litellm_core_utils/core_helpers.py). That function returnslitellm_metadatawhenever it is non-empty, afteradd_missing_spend_metadata_to_litellm_metadatahas copied over only the keys containinguser_api_key. Sospend_logs_metadatais dropped there, even thoughkwargs["litellm_params"]["metadata"]["spend_logs_metadata"]andstandard_logging_object["metadata"]["spend_logs_metadata"]both still hold it at that point.
Candidate fix, verified locally: route /interactions like /responses.
LITELLM_METADATA_ROUTES: Final = (
"batches",
"bedrock",
"/v1/messages",
"responses",
"files",
+ "interactions",
)
With that line, the header and body litellm_metadata are both recorded on /v1beta/interactions; body metadata is not, which matches the /v1/responses convention. Chat completions are unchanged. A broader alternative would be to have add_missing_spend_metadata_to_litellm_metadata also carry over spend_logs_metadata (as #47790 did for litellm_received_at), which would cover any other route where the two dicts split.
Related: #34905 (the same merge helper drops the router's model_group when a guardrail plants litellm_metadata), #47790 (merged 2026-10-21; carries litellm_received_at across the same merge, not spend_logs_metadata), #39749 (store: false rows dropped on this route, which is why the repro below uses store: true).
Config
model_list:
- model_name: gemini-3.1-flash-lite
litellm_params:
model: gemini/gemini-3.1-flash-lite
api_key: os.environ/GEMINI_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
LiteLLM Version
v1.106.1
Steps to Repro
Proxy: official ghcr.io/berriai/litellm:v1.106.1 image with Postgres and the config above.
- Request:
bash curl -X POST http://localhost:4000/v1beta/interactions \ -H "Authorization: Bearer sk-..." -H "Content-Type: application/json" \ -H "x-litellm-session-id: probe-session" \ -H 'x-litellm-spend-logs-metadata: {"probe": "header"}' \ -d '{"model": "gemini-3.1-flash-lite", "input": "Reply with the single word OK.", "store": true}'Response: 200 with the interaction object (store: trueso that #39749 cannot drop the row). - The same request with body
"metadata": {"spend_logs_metadata": {"probe": "body-metadata"}}instead of the header. - The same request with body
"litellm_metadata": {"spend_logs_metadata": {"probe": "body-litellm-metadata"}}. - The same three on
/v1/chat/completions("messages": [{"role": "user", "content": "Reply with the single word OK."}]). SELECT call_type, metadata->>'spend_logs_metadata' FROM "LiteLLM_SpendLogs" WHERE session_id = 'probe-session';
All requests return 200. Results:
| Route | header | body metadata | body litellm_metadata |
|---|---|---|---|
/v1/chat/completions | recorded | recorded | recorded |
/v1beta/interactions, v1.106.1 | null | null | null |
/v1beta/interactions, v1.106.1 + "interactions" in LITELLM_METADATA_ROUTES | recorded | null | recorded |
Which part of LiteLLM is this about?
Spend tracking: spend logs, cost attribution, usage reports
How are you deploying?
Docker
The two patches
litellm/litellm_core_utils/core_helpers.pyindex 1d61ef8519..48273b5e54 100644@@ -331,12 +331,22 @@ def add_missing_spend_metadata_to_litellm_metadata(litellm_metadata: dict, metad PATCH for issue where both `litellm_metadata` and `metadata` are present in the kwargs and user_api_key values are in 'metadata'. The proxy receive stamp rides along for the same reason: a guardrail hook can plant `litellm_metadata` on a chat request after the proxy stamped- `metadata`, and the response timing headers anchor on that stamp.+ `metadata`, and the response timing headers anchor on that stamp. `spend_logs_metadata` rides+ along when `metadata` is the proxy's request bucket (it holds the `user_api_key` stamps), since+ the router plants its deployment info in `litellm_metadata` on generic API routes and the spend+ log reads only the merged dict. """ potential_spend_tracking_metadata_substring: Final = "user_api_key"- for key, value in dict(metadata).items():+ request_metadata: Final = dict(metadata)+ is_proxy_request_metadata: Final = any(+ potential_spend_tracking_metadata_substring in key for key in request_metadata+ )+ for key, value in request_metadata.items(): carry_receive_stamp = key == "litellm_received_at" and key not in litellm_metadata- if potential_spend_tracking_metadata_substring in key or carry_receive_stamp:+ carry_spend_logs_metadata = (+ key == "spend_logs_metadata" and is_proxy_request_metadata and key not in litellm_metadata+ )+ if potential_spend_tracking_metadata_substring in key or carry_receive_stamp or carry_spend_logs_metadata: litellm_metadata[key] = value return litellm_metadata litellm/proxy/litellm_pre_call_utils.pyindex 8a45f658f3..da849ee4fa 100644@@ -334,6 +334,7 @@ LITELLM_METADATA_ROUTES: Final = ( "/v1/messages", "responses", "files",+ "interactions", ) LITELLM_TRACE_CONTROL_METADATA_FIELDS: Final = frozenset(tests/unit/litellm_core_utils/test_core_helpers.pyindex d66c37ba8c..08b300866f 100644@@ -719,3 +719,41 @@ class TestReceiveStampSurvivesASecondMetadataBucket: } assert get_litellm_metadata_from_kwargs(kwargs)["litellm_received_at"] is own+++class TestSpendLogsMetadataSurvivesASecondMetadataBucket:+ """The router plants deployment info in ``litellm_metadata`` on generic API routes whose proxy+ request metadata lives in ``metadata``; the spend log reads only the resolved bucket."""++ def test_spend_logs_metadata_is_carried_from_proxy_request_metadata(self):+ kwargs = {+ "litellm_params": {+ "metadata": {"user_api_key_hash": "hash", "spend_logs_metadata": {"probe": "header"}},+ "litellm_metadata": {"model_group": "gemini-3.1-flash-lite"},+ }+ }++ resolved = get_litellm_metadata_from_kwargs(kwargs)++ assert resolved["spend_logs_metadata"] == {"probe": "header"}+ assert resolved["model_group"] == "gemini-3.1-flash-lite"++ def test_litellm_metadata_keeps_its_own_spend_logs_metadata(self):+ kwargs = {+ "litellm_params": {+ "metadata": {"user_api_key_hash": "hash", "spend_logs_metadata": {"probe": "metadata"}},+ "litellm_metadata": {"spend_logs_metadata": {"probe": "litellm-metadata"}},+ }+ }++ assert get_litellm_metadata_from_kwargs(kwargs)["spend_logs_metadata"] == {"probe": "litellm-metadata"}++ def test_provider_body_metadata_spend_logs_metadata_is_not_carried(self):+ kwargs = {+ "litellm_params": {+ "metadata": {"spend_logs_metadata": {"probe": "provider-metadata"}},+ "litellm_metadata": {"user_api_key_hash": "hash"},+ }+ }++ assert "spend_logs_metadata" not in get_litellm_metadata_from_kwargs(kwargs)tests/unit/proxy/test_litellm_pre_call_utils.pyindex d9def23bde..5e6bc829cd 100644@@ -10537,3 +10537,102 @@ def test_mapped_baggage_tags_keep_non_string_client_tags_as_sent(): data: Final[dict[str, object]] = {"metadata": {"tags": [1, {"team": "x"}, "client-tag", "client-tag"]}} LiteLLMProxyRequestSetup.apply_authenticated_baggage_tags(request, data) assert data["metadata"]["tags"] == [1, {"team": "x"}, "client-tag", "client-tag", "tenant:a"]+++_ROUTER_DEPLOYMENT_METADATA: Final = {+ "model_group": "gemini-3.1-flash-lite",+ "deployment": "gemini/gemini-3.1-flash-lite",+ "model_info": {"id": "deployment-1"},+}+++def _spend_log_row_spend_logs_metadata(proxy_data: dict) -> object:+ """Build the LiteLLM_SpendLogs row after the router planted its deployment info in litellm_metadata."""+ litellm_metadata = dict(proxy_data.get("litellm_metadata") or {})+ litellm_metadata.update(_ROUTER_DEPLOYMENT_METADATA)+ row = get_logging_payload(+ kwargs={+ "model": "gemini-3.1-flash-lite",+ "custom_llm_provider": "gemini",+ "litellm_params": {+ "metadata": dict(proxy_data.get("metadata") or {}),+ "litellm_metadata": litellm_metadata,+ },+ },+ response_obj={},+ start_time=datetime.now(timezone.utc),+ end_time=datetime.now(timezone.utc),+ )+ return json.loads(row["metadata"])["spend_logs_metadata"]+++@pytest.mark.asyncio+@pytest.mark.parametrize("path", ["/v1beta/interactions", "/interactions"])+async def test_interactions_route_records_spend_logs_metadata_from_header(path: str):+ updated = await add_litellm_data_to_request(+ data={"model": "gemini-3.1-flash-lite", "input": "hi", "store": True},+ request=_make_request_mock(+ path, {"Content-Type": "application/json", "x-litellm-spend-logs-metadata": '{"probe": "header"}'}+ ),+ user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),+ proxy_config=MagicMock(),+ general_settings={},+ version="test-version",+ )++ assert updated["litellm_metadata"]["spend_logs_metadata"] == {"probe": "header"}+ assert _spend_log_row_spend_logs_metadata(updated) == {"probe": "header"}+++@pytest.mark.asyncio+@pytest.mark.parametrize("path", ["/v1beta/interactions", "/interactions"])+async def test_interactions_route_records_spend_logs_metadata_from_body_litellm_metadata(path: str):+ updated = await add_litellm_data_to_request(+ data={+ "model": "gemini-3.1-flash-lite",+ "input": "hi",+ "store": True,+ "litellm_metadata": {"spend_logs_metadata": {"probe": "body-litellm-metadata"}},+ },+ request=_make_request_mock(path, {"Content-Type": "application/json"}),+ user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),+ proxy_config=MagicMock(),+ general_settings={},+ version="test-version",+ )++ assert _spend_log_row_spend_logs_metadata(updated) == {"probe": "body-litellm-metadata"}+++@pytest.mark.asyncio+@pytest.mark.parametrize("path", ["/v1/videos", "/v1/vector_stores", "/v1/containers"])+async def test_generic_api_routes_record_spend_logs_metadata_after_router_plants_litellm_metadata(path: str):+ """These routes keep request metadata in ``metadata`` while the router's generic API call+ plants deployment info in ``litellm_metadata``; the spend log must still see the header."""+ updated = await add_litellm_data_to_request(+ data={"model": "gemini-3.1-flash-lite"},+ request=_make_request_mock(+ path, {"Content-Type": "application/json", "x-litellm-spend-logs-metadata": '{"probe": "header"}'}+ ),+ user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),+ proxy_config=MagicMock(),+ general_settings={},+ version="test-version",+ )++ assert updated["metadata"]["spend_logs_metadata"] == {"probe": "header"}+ assert _spend_log_row_spend_logs_metadata(updated) == {"probe": "header"}+++@pytest.mark.asyncio+async def test_responses_route_does_not_record_spend_logs_metadata_from_provider_body_metadata():+ updated = await add_litellm_data_to_request(+ data={"model": "gpt-4o", "input": "hi", "metadata": {"spend_logs_metadata": {"probe": "body-metadata"}}},+ request=_make_request_mock("/v1/responses", {"Content-Type": "application/json"}),+ user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),+ proxy_config=MagicMock(),+ general_settings={},+ version="test-version",+ )++ assert _spend_log_row_spend_logs_metadata(updated) is None
litellm/proxy/litellm_pre_call_utils.pyindex 8a45f658f36..da849ee4fad 100644@@ -334,6 +334,7 @@ LITELLM_METADATA_ROUTES: Final = ( "/v1/messages", "responses", "files",+ "interactions", ) LITELLM_TRACE_CONTROL_METADATA_FIELDS: Final = frozenset(tests/unit/proxy/test_litellm_pre_call_utils.pyindex d9def23bde8..04f6381f32f 100644@@ -106,6 +106,13 @@ class TestGetMetadataVariableName: request = self._make_request("/v1/files") assert get_metadata_variable_name(request) == "litellm_metadata" + @pytest.mark.parametrize(+ "path", ["/v1beta/interactions", "/interactions", "/v1beta/interactions/v1_abc123", "/interactions/v1_abc123/cancel"]+ )+ def test_returns_litellm_metadata_for_interactions_routes(self, path):+ request = self._make_request(path)+ assert get_metadata_variable_name(request) == "litellm_metadata"+ def test_returns_metadata_for_chat_completions(self): request = self._make_request("/chat/completions") assert get_metadata_variable_name(request) == "metadata"@@ -708,6 +715,46 @@ async def test_add_litellm_data_to_request_parses_json_object_string_litellm_met assert updated["litellm_metadata"]["cost_centre"] == "research" +@pytest.mark.asyncio+async def test_interactions_spend_logs_metadata_survives_the_router_metadata_bucket():+ """GH#47841: on /v1beta/interactions the router plants its deployment fields in+ `litellm_metadata`. The proxy must put the request's metadata there too: otherwise the+ spend log reads only the router's bucket and `spend_logs_metadata` from the+ `x-litellm-spend-logs-metadata` header never reaches LiteLLM_SpendLogs."""+ request_mock = MagicMock(spec=Request)+ request_mock.url = MagicMock()+ request_mock.url.__str__.return_value = "http://localhost/v1beta/interactions"+ request_mock.url.path = "/v1beta/interactions"+ request_mock.method = "POST"+ request_mock.query_params = {}+ request_mock.headers = {+ "Content-Type": "application/json",+ "x-litellm-spend-logs-metadata": '{"cost_centre": "research"}',+ }+ request_mock.client = MagicMock()+ request_mock.client.host = "127.0.0.1"+ request_mock.state.parent_otel_span = None++ data = await add_litellm_data_to_request(+ data={"model": "gemini-3.1-flash-lite", "input": "Reply OK.", "store": True},+ request=request_mock,+ user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),+ proxy_config=MagicMock(),+ general_settings={},+ version="test-version",+ )+ assert data["litellm_metadata"]["spend_logs_metadata"] == {"cost_centre": "research"}++ # The router then adds its deployment fields to the same bucket, and the spend log+ # resolves that bucket.+ data["litellm_metadata"].update({"model_group": "gemini-3.1-flash-lite", "deployment": "gemini/gemini-3.1-flash-lite"})+ resolved = get_litellm_metadata_from_kwargs(+ {"litellm_params": {"metadata": data.get("metadata") or {}, "litellm_metadata": data["litellm_metadata"]}}+ )+ assert resolved["spend_logs_metadata"] == {"cost_centre": "research"}+ assert resolved["model_group"] == "gemini-3.1-flash-lite"++ @pytest.mark.asyncio async def test_add_litellm_data_to_request_proxy_server_request_body_is_post_strip(): """Regression: proxy_server_request['body'] used to be snapshotted before