Backtestslitellmrun-20261019-033420#25857

Issue #25857 Merged fix PR # Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: amazon.titan-embed-image-v1 — usage.prompt_tokens=0 for image inputs (EmbeddingResponse field, not cost calc)

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

The functional change is identical in both patches: prompt_tokens and total_tokens become total_prompt_tokens + image_count, exactly as the issue proposes, and the independent tests confirm both fix the reported behaviour equally (all five pass on A and B, fail on base). The difference is in tests and craft. Patch A only appends two assertions to an existing cost-tracking test, covering the single-image case. Patch B adds four dedicated tests: image-only batch of two, mixed text+image (verifying the sum 5+1=6), a text-only regression guard, and an end-to-end litellm.embedding() test that also checks completion_cost still equals input_cost_per_image, directly guarding the issue's concern that the change must not alter cost calculation. B also adds a concise comment explaining why images count as one unit and why cost is unaffected, which matters since the code otherwise looks like it conflates tokens with images. Neither patch touches the Nova transformation the issue flags as 'worth checking', so scope is equal. B wins on test quality and clarity with no added risk.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

15m 59s

Wall-clock time in the container

Full test suite

No new failures

30 passed · 0 skipped · 30 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

118vs 6 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When calling litellm.embedding() with amazon.titan-embed-image-v1 and an image input (data URI), the returned EmbeddingResponse.usage.prompt_tokens is always 0, even though the model successfully processed the image and returned an embedding.

This is distinct from the cost-calculation bug fixed in PR #21646 (merged Feb 20, 2026). That PR correctly fixed LiteLLM's internal cost calculator to use input_cost_per_image via prompt_tokens_details.image_count. However, usage.prompt_tokens itself is still 0 in the returned response object.

Any downstream consumer (e.g., a quota/billing system, logging layer, or monitoring tool) reading response.usage.prompt_tokens directly — rather than going through LiteLLM's cost calculator — gets 0 and cannot detect that work was performed.


Root cause

In litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.py, _transform_response():

total_prompt_tokens += _parsed_response["inputTextTokenCount"]  # always 0 for image inputs

# image_count IS correctly counted (fix from #21646):
if image_count > 0:
    prompt_tokens_details = PromptTokensDetailsWrapper(image_count=image_count)

# But prompt_tokens is set from total_prompt_tokens, which is still 0:
usage = Usage(
    prompt_tokens=total_prompt_tokens,   # <- 0 for image-only input
    total_tokens=total_prompt_tokens,    # <- 0
    prompt_tokens_details=prompt_tokens_details,
)

AWS returns inputTextTokenCount=0 for image inputs by design — amazon.titan-embed-image-v1 charges per image at a flat rate of $0.00006/image, not per token. LiteLLM's pricing registry already encodes this correctly:

"amazon.titan-embed-image-v1": {
    "input_cost_per_image": 6e-05,
    "input_cost_per_token": 8e-07
}

PR #21646 fixed the cost calculator path (generic_cost_per_token in utils.py now reads image_count from prompt_tokens_details and applies input_cost_per_image). But the usage.prompt_tokens field on the EmbeddingResponse was never updated, so it remains 0.


Expected vs. actual

FieldExpectedActual (v1.83.8)
usage.prompt_tokensN (number of images, e.g. 1)0 ← bug
usage.total_tokensN0 ← bug
usage.prompt_tokens_details.image_countNcorrect (fixed by #21646)
litellm.completion_cost(response)correct ($0.00006)correct (fixed by #21646)

Proposed fix

In _transform_response() of amazon_titan_multimodal_transformation.py:

prompt_tokens_details = None
if image_count > 0:
    prompt_tokens_details = PromptTokensDetailsWrapper(
        image_count=image_count,
        image_tokens=image_count,  # populate the existing but currently unused field
    )

usage = Usage(
    prompt_tokens=total_prompt_tokens + image_count,   # non-zero when images were processed
    completion_tokens=0,
    total_tokens=total_prompt_tokens + image_count,
    prompt_tokens_details=prompt_tokens_details,
)

This ensures prompt_tokens > 0 for any completed image embedding call and is fully backward-compatible (text-only paths unaffected since image_count=0).

The same gap likely exists in amazon_nova_transformation.py — worth checking AmazonNovaEmbeddingConfig._transform_response() for identical behaviour.


Related

  • PR #21646 — fixed cost calculation, but not usage.prompt_tokens

Steps to Reproduce

Minimal pure-LiteLLM repro (no real AWS call needed for the logic, but an AWS account is needed to run):

import base64
from io import BytesIO
from PIL import Image
from litellm import embedding

# Image must be a data URI — litellm's is_base64_encoded() requires `data:` prefix
img = Image.new("RGB", (8, 8), color="blue")
buf = BytesIO()
img.save(buf, format="PNG")
img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
img_data_uri = f"data:image/png;base64,{img_b64}"

response = embedding(
    model="amazon.titan-embed-image-v1",
    input=[img_data_uri],
    aws_region_name="us-east-1",
)

print(response.usage)
# Usage(prompt_tokens=0, total_tokens=0, ...)  ← BUG: should be non-zero

print(response.usage.prompt_tokens_details)
# PromptTokensDetailsWrapper(image_count=1)  ← image IS tracked here, just not surfaced in prompt_tokens

Raw boto3 confirmation (proves this is a LiteLLM-level issue, not AWS):

import json, base64, boto3
from io import BytesIO
from PIL import Image

img = Image.new("RGB", (8, 8), color="blue")
buf = BytesIO()
img.save(buf, format="PNG")
img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
resp = bedrock.invoke_model(
    modelId="amazon.titan-embed-image-v1",
    body=json.dumps({"inputImage": img_b64}),
    accept="application/json",
    contentType="application/json",
)

print(json.loads(resp["body"].read()))
# {"embedding": [...], "inputTextTokenCount": 0}
# AWS intentionally returns inputTextTokenCount=0 for images —
# the model is priced PER IMAGE ($0.00006/image), not per token.
# There is no "inputImageTokenCount" field — AWS does not expose one.

Relevant log output

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.83.8

Twitter / LinkedIn details

https://www.linkedin.com/in/shay-margalit-40581668/

The two patches

Our patch+116 −2
litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.pyindex 07b04734c3..a42207c240 100644@@ -92,10 +92,16 @@ class AmazonTitanMultimodalEmbeddingG1Config:                 image_count=image_count,             ) +        # AWS returns inputTextTokenCount=0 for image inputs (images are billed+        # per image, not per token), so count each image as one prompt unit to+        # keep usage.prompt_tokens non-zero. Cost calc uses image_count with+        # input_cost_per_image, so this does not affect the computed cost.+        prompt_tokens = total_prompt_tokens + image_count+         usage = Usage(-            prompt_tokens=total_prompt_tokens,+            prompt_tokens=prompt_tokens,             completion_tokens=0,-            total_tokens=total_prompt_tokens,+            total_tokens=prompt_tokens,             prompt_tokens_details=prompt_tokens_details,         )         return EmbeddingResponse(model=model, usage=usage, data=transformed_responses)tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.pyindex a38a6612f7..b0a2f7f236 100644@@ -955,3 +955,111 @@ def test_titan_image_embedding_cost_uses_per_image_rate():         assert response.usage is not None         assert response.usage.prompt_tokens_details is not None         assert response.usage.prompt_tokens_details.image_count == 1+++def test_titan_multimodal_embedding_image_prompt_tokens_non_zero():+    """+    Image-only Titan multimodal embeddings must report non-zero usage.prompt_tokens.+    AWS returns inputTextTokenCount=0 for images, so each image counts as one unit.+    """+    from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+        AmazonTitanMultimodalEmbeddingG1Config,+    )++    config = AmazonTitanMultimodalEmbeddingG1Config()++    result = config._transform_response(+        response_list=[+            {"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 0},+            {"embedding": [0.4, 0.5, 0.6], "inputTextTokenCount": 0},+        ],+        model="amazon.titan-embed-image-v1",+        batch_data=[+            {"inputImage": "/9j/4AAQSkZJRg=="},+            {"inputImage": "/9j/4AAQSkZJRg=="},+        ],+    )++    assert result.usage.prompt_tokens == 2+    assert result.usage.total_tokens == 2+    assert result.usage.prompt_tokens_details.image_count == 2+++def test_titan_multimodal_embedding_mixed_text_and_image_prompt_tokens():+    """Mixed text + image batch: prompt_tokens = text tokens + image count."""+    from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+        AmazonTitanMultimodalEmbeddingG1Config,+    )++    config = AmazonTitanMultimodalEmbeddingG1Config()++    result = config._transform_response(+        response_list=[+            {"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 5},+            {"embedding": [0.4, 0.5, 0.6], "inputTextTokenCount": 0},+        ],+        model="amazon.titan-embed-image-v1",+        batch_data=[+            {"inputText": "hello world"},+            {"inputImage": "/9j/4AAQSkZJRg=="},+        ],+    )++    assert result.usage.prompt_tokens == 6+    assert result.usage.total_tokens == 6+    assert result.usage.prompt_tokens_details.image_count == 1+++def test_titan_multimodal_embedding_text_only_prompt_tokens_unchanged():+    """Text-only Titan multimodal embeddings keep prompt_tokens = inputTextTokenCount."""+    from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+        AmazonTitanMultimodalEmbeddingG1Config,+    )++    config = AmazonTitanMultimodalEmbeddingG1Config()++    result = config._transform_response(+        response_list=[{"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 5}],+        model="amazon.titan-embed-image-v1",+        batch_data=[{"inputText": "hello world"}],+    )++    assert result.usage.prompt_tokens == 5+    assert result.usage.total_tokens == 5+++def test_titan_image_embedding_e2e_prompt_tokens_and_cost():+    """+    End-to-end: litellm.embedding() with an image data URI on+    amazon.titan-embed-image-v1 returns non-zero usage.prompt_tokens, and the+    cost is still exactly one image at input_cost_per_image (no extra token charge).+    """+    client = HTTPHandler()++    with patch.object(client, "post") as mock_post:+        mock_response = Mock()+        mock_response.status_code = 200+        mock_response.text = json.dumps(+            {"embedding": [0.1] * 1024, "inputTextTokenCount": 0}+        )+        mock_response.json = lambda: json.loads(mock_response.text)+        mock_post.return_value = mock_response++        response = litellm.embedding(+            model="bedrock/amazon.titan-embed-image-v1",+            input=["data:image/png;base64,iVBORw0KGgoAAAANSUhEUg=="],+            client=client,+            aws_access_key_id="fake",+            aws_secret_access_key="fake",+            aws_region_name="us-east-1",+        )++    assert response.usage.prompt_tokens == 1+    assert response.usage.total_tokens == 1+    assert response.usage.prompt_tokens_details.image_count == 1++    model_info = litellm.get_model_info(+        model="amazon.titan-embed-image-v1", custom_llm_provider="bedrock"+    )+    cost = litellm.completion_cost(completion_response=response)+    assert cost == pytest.approx(model_info["input_cost_per_image"]) 
Merged fix · PR #+4 −2
litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.pyindex 07b04734c30..d67f936aa88 100644@@ -93,9 +93,9 @@ class AmazonTitanMultimodalEmbeddingG1Config:             )          usage = Usage(-            prompt_tokens=total_prompt_tokens,+            prompt_tokens=total_prompt_tokens + image_count,             completion_tokens=0,-            total_tokens=total_prompt_tokens,+            total_tokens=total_prompt_tokens + image_count,             prompt_tokens_details=prompt_tokens_details,         )         return EmbeddingResponse(model=model, usage=usage, data=transformed_responses)tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.pyindex a38a6612f79..d2813ecb3a8 100644@@ -863,6 +863,8 @@ def test_titan_multimodal_embedding_image_cost_tracking():     )      assert result.usage is not None+    assert result.usage.prompt_tokens == 1  # image_count included in prompt_tokens+    assert result.usage.total_tokens == 1     assert result.usage.prompt_tokens_details is not None     assert result.usage.prompt_tokens_details.image_count == 1