Backtestslitellmrun-20261018-182052#25857

Issue #25857 Merged fix PR # Run 18 Oct 2026 · run-20261018-182052 BerriAI/litellm

[Bug]: amazon.titan-embed-image-v1 — usage.prompt_tokens=0 for image inputs (EmbeddingResponse field, not cost calc)

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches make the same core change: usage.prompt_tokens and total_tokens become total_prompt_tokens + image_count, which directly resolves the reported zero-usage symptom for image inputs. The difference is in the follow-on cost-calculation interaction and in test quality. Patch A additionally populates prompt_tokens_details.text_tokens with the real text-token count whenever images are present. LiteLLM's generic cost path prefers text_tokens from prompt_tokens_details when set, so in a mixed text+image batch Patch A keeps the per-token charge at the true text count (5 tokens) rather than letting the synthetic image units inflate it; Patch B leaves text_tokens unset, so the calculator falls back to prompt_tokens (6) and overcharges per-token cost for mixed inputs, a regression introduced by its own change. Patch A's tests are also substantially stronger: an image-only unit test asserting prompt_tokens==2 plus cost==2*6e-05, a mixed-batch test asserting text_tokens, image_count, and the combined cost, and an end-to-end litellm.embedding() test with a mocked HTTP client asserting prompt_tokens==1 and cost. These would fail before the change and explicitly guard the invariant the issue cares about (usage non-zero while cost stays per-image). Patch B only adds two assertions to an existing test and never checks that cost is unaffected by the new prompt_tokens value. Neither patch touches the Nova transformation the issue flagged as a likely duplicate, so that is a wash. Patch A's extra text_tokens line is minimal, commented, and directly tied to preventing cost drift from the fix, so it is not scope creep. Patch A is the better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

11m 33s

Wall-clock time in the container

Full test suite

No new failures

29 passed · 0 skipped · 29 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

107vs 6 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When calling litellm.embedding() with amazon.titan-embed-image-v1 and an image input (data URI), the returned EmbeddingResponse.usage.prompt_tokens is always 0, even though the model successfully processed the image and returned an embedding.

This is distinct from the cost-calculation bug fixed in PR #21646 (merged Feb 20, 2026). That PR correctly fixed LiteLLM's internal cost calculator to use input_cost_per_image via prompt_tokens_details.image_count. However, usage.prompt_tokens itself is still 0 in the returned response object.

Any downstream consumer (e.g., a quota/billing system, logging layer, or monitoring tool) reading response.usage.prompt_tokens directly — rather than going through LiteLLM's cost calculator — gets 0 and cannot detect that work was performed.


Root cause

In litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.py, _transform_response():

total_prompt_tokens += _parsed_response["inputTextTokenCount"]  # always 0 for image inputs

# image_count IS correctly counted (fix from #21646):
if image_count > 0:
    prompt_tokens_details = PromptTokensDetailsWrapper(image_count=image_count)

# But prompt_tokens is set from total_prompt_tokens, which is still 0:
usage = Usage(
    prompt_tokens=total_prompt_tokens,   # <- 0 for image-only input
    total_tokens=total_prompt_tokens,    # <- 0
    prompt_tokens_details=prompt_tokens_details,
)

AWS returns inputTextTokenCount=0 for image inputs by design — amazon.titan-embed-image-v1 charges per image at a flat rate of $0.00006/image, not per token. LiteLLM's pricing registry already encodes this correctly:

"amazon.titan-embed-image-v1": {
    "input_cost_per_image": 6e-05,
    "input_cost_per_token": 8e-07
}

PR #21646 fixed the cost calculator path (generic_cost_per_token in utils.py now reads image_count from prompt_tokens_details and applies input_cost_per_image). But the usage.prompt_tokens field on the EmbeddingResponse was never updated, so it remains 0.


Expected vs. actual

FieldExpectedActual (v1.83.8)
usage.prompt_tokensN (number of images, e.g. 1)0 ← bug
usage.total_tokensN0 ← bug
usage.prompt_tokens_details.image_countNcorrect (fixed by #21646)
litellm.completion_cost(response)correct ($0.00006)correct (fixed by #21646)

Proposed fix

In _transform_response() of amazon_titan_multimodal_transformation.py:

prompt_tokens_details = None
if image_count > 0:
    prompt_tokens_details = PromptTokensDetailsWrapper(
        image_count=image_count,
        image_tokens=image_count,  # populate the existing but currently unused field
    )

usage = Usage(
    prompt_tokens=total_prompt_tokens + image_count,   # non-zero when images were processed
    completion_tokens=0,
    total_tokens=total_prompt_tokens + image_count,
    prompt_tokens_details=prompt_tokens_details,
)

This ensures prompt_tokens > 0 for any completed image embedding call and is fully backward-compatible (text-only paths unaffected since image_count=0).

The same gap likely exists in amazon_nova_transformation.py — worth checking AmazonNovaEmbeddingConfig._transform_response() for identical behaviour.


Related

  • PR #21646 — fixed cost calculation, but not usage.prompt_tokens

Steps to Reproduce

Minimal pure-LiteLLM repro (no real AWS call needed for the logic, but an AWS account is needed to run):

import base64
from io import BytesIO
from PIL import Image
from litellm import embedding

# Image must be a data URI — litellm's is_base64_encoded() requires `data:` prefix
img = Image.new("RGB", (8, 8), color="blue")
buf = BytesIO()
img.save(buf, format="PNG")
img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
img_data_uri = f"data:image/png;base64,{img_b64}"

response = embedding(
    model="amazon.titan-embed-image-v1",
    input=[img_data_uri],
    aws_region_name="us-east-1",
)

print(response.usage)
# Usage(prompt_tokens=0, total_tokens=0, ...)  ← BUG: should be non-zero

print(response.usage.prompt_tokens_details)
# PromptTokensDetailsWrapper(image_count=1)  ← image IS tracked here, just not surfaced in prompt_tokens

Raw boto3 confirmation (proves this is a LiteLLM-level issue, not AWS):

import json, base64, boto3
from io import BytesIO
from PIL import Image

img = Image.new("RGB", (8, 8), color="blue")
buf = BytesIO()
img.save(buf, format="PNG")
img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
resp = bedrock.invoke_model(
    modelId="amazon.titan-embed-image-v1",
    body=json.dumps({"inputImage": img_b64}),
    accept="application/json",
    contentType="application/json",
)

print(json.loads(resp["body"].read()))
# {"embedding": [...], "inputTextTokenCount": 0}
# AWS intentionally returns inputTextTokenCount=0 for images —
# the model is priced PER IMAGE ($0.00006/image), not per token.
# There is no "inputImageTokenCount" field — AWS does not expose one.

Relevant log output

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.83.8

Twitter / LinkedIn details

https://www.linkedin.com/in/shay-margalit-40581668/

The two patches

Our patch+105 −2
litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.pyindex 07b04734c3..17da5b30f7 100644@@ -88,14 +88,19 @@ class AmazonTitanMultimodalEmbeddingG1Config:          prompt_tokens_details: Optional[PromptTokensDetailsWrapper] = None         if image_count > 0:+            # Bedrock returns inputTextTokenCount=0 for images (billed per image),+            # so text tokens are kept separate from the image count for cost calc.             prompt_tokens_details = PromptTokensDetailsWrapper(+                text_tokens=total_prompt_tokens,                 image_count=image_count,             ) +        # Count each image as one prompt unit so usage is non-zero for image inputs+        prompt_tokens = total_prompt_tokens + image_count         usage = Usage(-            prompt_tokens=total_prompt_tokens,+            prompt_tokens=prompt_tokens,             completion_tokens=0,-            total_tokens=total_prompt_tokens,+            total_tokens=prompt_tokens,             prompt_tokens_details=prompt_tokens_details,         )         return EmbeddingResponse(model=model, usage=usage, data=transformed_responses)tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.pyindex a38a6612f7..7d312a4b3a 100644@@ -955,3 +955,101 @@ def test_titan_image_embedding_cost_uses_per_image_rate():         assert response.usage is not None         assert response.usage.prompt_tokens_details is not None         assert response.usage.prompt_tokens_details.image_count == 1+++def test_titan_multimodal_embedding_image_prompt_tokens_non_zero():+    """+    Image-only Titan multimodal embeddings must report non-zero usage.prompt_tokens,+    since Bedrock returns inputTextTokenCount=0 for images.+    """+    from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+        AmazonTitanMultimodalEmbeddingG1Config,+    )++    config = AmazonTitanMultimodalEmbeddingG1Config()++    result = config._transform_response(+        response_list=[+            {"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 0},+            {"embedding": [0.4, 0.5, 0.6], "inputTextTokenCount": 0},+        ],+        model="amazon.titan-embed-image-v1",+        batch_data=[+            {"inputImage": "/9j/4AAQSkZJRg=="},+            {"inputImage": "iVBORw0KGgo="},+        ],+    )++    assert result.usage.prompt_tokens == 2+    assert result.usage.total_tokens == 2+    assert result.usage.prompt_tokens_details.image_count == 2++    # cost must stay per-image only, not inflated by the image units in prompt_tokens+    cost = litellm.completion_cost(+        completion_response=result,+        model="bedrock/amazon.titan-embed-image-v1",+        call_type="embedding",+    )+    assert cost == pytest.approx(2 * 6e-05)+++def test_titan_multimodal_embedding_mixed_text_and_image_usage():+    """Mixed text+image batch: prompt_tokens includes text tokens and images; cost charges both."""+    from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+        AmazonTitanMultimodalEmbeddingG1Config,+    )++    config = AmazonTitanMultimodalEmbeddingG1Config()++    result = config._transform_response(+        response_list=[+            {"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 5},+            {"embedding": [0.4, 0.5, 0.6], "inputTextTokenCount": 0},+        ],+        model="amazon.titan-embed-image-v1",+        batch_data=[+            {"inputText": "hello world"},+            {"inputImage": "/9j/4AAQSkZJRg=="},+        ],+    )++    assert result.usage.prompt_tokens == 6+    assert result.usage.total_tokens == 6+    assert result.usage.prompt_tokens_details.text_tokens == 5+    assert result.usage.prompt_tokens_details.image_count == 1++    cost = litellm.completion_cost(+        completion_response=result,+        model="bedrock/amazon.titan-embed-image-v1",+        call_type="embedding",+    )+    assert cost == pytest.approx(5 * 8e-07 + 6e-05)+++def test_titan_image_embedding_e2e_prompt_tokens_non_zero():+    """End-to-end: litellm.embedding() with an image data URI returns prompt_tokens > 0."""+    client = HTTPHandler()++    with patch.object(client, "post") as mock_post:+        mock_response = Mock()+        mock_response.status_code = 200+        mock_response.text = json.dumps(+            {"embedding": [0.1] * 1024, "inputTextTokenCount": 0}+        )+        mock_response.json = lambda: json.loads(mock_response.text)+        mock_post.return_value = mock_response++        response = litellm.embedding(+            model="bedrock/amazon.titan-embed-image-v1",+            input=["data:image/png;base64,iVBORw0KGgoAAAANSUhEUg=="],+            client=client,+            aws_access_key_id="fake",+            aws_secret_access_key="fake",+            aws_region_name="us-east-1",+        )++        assert response.usage.prompt_tokens == 1+        assert response.usage.total_tokens == 1+        assert litellm.completion_cost(completion_response=response) == pytest.approx(+            6e-05+        ) 
Merged fix · PR #+4 −2
litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.pyindex 07b04734c30..d67f936aa88 100644@@ -93,9 +93,9 @@ class AmazonTitanMultimodalEmbeddingG1Config:             )          usage = Usage(-            prompt_tokens=total_prompt_tokens,+            prompt_tokens=total_prompt_tokens + image_count,             completion_tokens=0,-            total_tokens=total_prompt_tokens,+            total_tokens=total_prompt_tokens + image_count,             prompt_tokens_details=prompt_tokens_details,         )         return EmbeddingResponse(model=model, usage=usage, data=transformed_responses)tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.pyindex a38a6612f79..d2813ecb3a8 100644@@ -863,6 +863,8 @@ def test_titan_multimodal_embedding_image_cost_tracking():     )      assert result.usage is not None+    assert result.usage.prompt_tokens == 1  # image_count included in prompt_tokens+    assert result.usage.total_tokens == 1     assert result.usage.prompt_tokens_details is not None     assert result.usage.prompt_tokens_details.image_count == 1