Backtestslitellmrun-20261018-182052#25857
[Bug]: amazon.titan-embed-image-v1 — usage.prompt_tokens=0 for image inputs (EmbeddingResponse field, not cost calc)
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches make the same core change: usage.prompt_tokens and total_tokens become total_prompt_tokens + image_count, which directly resolves the reported zero-usage symptom for image inputs. The difference is in the follow-on cost-calculation interaction and in test quality. Patch A additionally populates prompt_tokens_details.text_tokens with the real text-token count whenever images are present. LiteLLM's generic cost path prefers text_tokens from prompt_tokens_details when set, so in a mixed text+image batch Patch A keeps the per-token charge at the true text count (5 tokens) rather than letting the synthetic image units inflate it; Patch B leaves text_tokens unset, so the calculator falls back to prompt_tokens (6) and overcharges per-token cost for mixed inputs, a regression introduced by its own change. Patch A's tests are also substantially stronger: an image-only unit test asserting prompt_tokens==2 plus cost==2*6e-05, a mixed-batch test asserting text_tokens, image_count, and the combined cost, and an end-to-end litellm.embedding() test with a mocked HTTP client asserting prompt_tokens==1 and cost. These would fail before the change and explicitly guard the invariant the issue cares about (usage non-zero while cost stays per-image). Patch B only adds two assertions to an existing test and never checks that cost is unaffected by the new prompt_tokens value. Neither patch touches the Nova transformation the issue flagged as a likely duplicate, so that is a wash. Patch A's extra text_tokens line is minimal, commented, and directly tied to preventing cost drift from the fix, so it is not scope creep. Patch A is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
11m 33s
Wall-clock time in the container
Full test suite
No new failures
29 passed · 0 skipped · 29 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
107vs 6 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
When calling litellm.embedding() with amazon.titan-embed-image-v1 and an image input (data URI), the returned EmbeddingResponse.usage.prompt_tokens is always 0, even though the model successfully processed the image and returned an embedding.
This is distinct from the cost-calculation bug fixed in PR #21646 (merged Feb 20, 2026). That PR correctly fixed LiteLLM's internal cost calculator to use input_cost_per_image via prompt_tokens_details.image_count. However, usage.prompt_tokens itself is still 0 in the returned response object.
Any downstream consumer (e.g., a quota/billing system, logging layer, or monitoring tool) reading response.usage.prompt_tokens directly — rather than going through LiteLLM's cost calculator — gets 0 and cannot detect that work was performed.
Root cause
In litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.py, _transform_response():
total_prompt_tokens += _parsed_response["inputTextTokenCount"] # always 0 for image inputs
# image_count IS correctly counted (fix from #21646):
if image_count > 0:
prompt_tokens_details = PromptTokensDetailsWrapper(image_count=image_count)
# But prompt_tokens is set from total_prompt_tokens, which is still 0:
usage = Usage(
prompt_tokens=total_prompt_tokens, # <- 0 for image-only input
total_tokens=total_prompt_tokens, # <- 0
prompt_tokens_details=prompt_tokens_details,
)
AWS returns inputTextTokenCount=0 for image inputs by design — amazon.titan-embed-image-v1 charges per image at a flat rate of $0.00006/image, not per token. LiteLLM's pricing registry already encodes this correctly:
"amazon.titan-embed-image-v1": {
"input_cost_per_image": 6e-05,
"input_cost_per_token": 8e-07
}
PR #21646 fixed the cost calculator path (generic_cost_per_token in utils.py now reads image_count from prompt_tokens_details and applies input_cost_per_image). But the usage.prompt_tokens field on the EmbeddingResponse was never updated, so it remains 0.
Expected vs. actual
| Field | Expected | Actual (v1.83.8) |
|---|---|---|
usage.prompt_tokens | N (number of images, e.g. 1) | 0 ← bug |
usage.total_tokens | N | 0 ← bug |
usage.prompt_tokens_details.image_count | N | correct (fixed by #21646) |
litellm.completion_cost(response) | correct ($0.00006) | correct (fixed by #21646) |
Proposed fix
In _transform_response() of amazon_titan_multimodal_transformation.py:
prompt_tokens_details = None
if image_count > 0:
prompt_tokens_details = PromptTokensDetailsWrapper(
image_count=image_count,
image_tokens=image_count, # populate the existing but currently unused field
)
usage = Usage(
prompt_tokens=total_prompt_tokens + image_count, # non-zero when images were processed
completion_tokens=0,
total_tokens=total_prompt_tokens + image_count,
prompt_tokens_details=prompt_tokens_details,
)
This ensures prompt_tokens > 0 for any completed image embedding call and is fully backward-compatible (text-only paths unaffected since image_count=0).
The same gap likely exists in amazon_nova_transformation.py — worth checking AmazonNovaEmbeddingConfig._transform_response() for identical behaviour.
Related
- PR #21646 — fixed cost calculation, but not
usage.prompt_tokens
Steps to Reproduce
Minimal pure-LiteLLM repro (no real AWS call needed for the logic, but an AWS account is needed to run):
import base64
from io import BytesIO
from PIL import Image
from litellm import embedding
# Image must be a data URI — litellm's is_base64_encoded() requires `data:` prefix
img = Image.new("RGB", (8, 8), color="blue")
buf = BytesIO()
img.save(buf, format="PNG")
img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
img_data_uri = f"data:image/png;base64,{img_b64}"
response = embedding(
model="amazon.titan-embed-image-v1",
input=[img_data_uri],
aws_region_name="us-east-1",
)
print(response.usage)
# Usage(prompt_tokens=0, total_tokens=0, ...) ← BUG: should be non-zero
print(response.usage.prompt_tokens_details)
# PromptTokensDetailsWrapper(image_count=1) ← image IS tracked here, just not surfaced in prompt_tokens
Raw boto3 confirmation (proves this is a LiteLLM-level issue, not AWS):
import json, base64, boto3
from io import BytesIO
from PIL import Image
img = Image.new("RGB", (8, 8), color="blue")
buf = BytesIO()
img.save(buf, format="PNG")
img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
resp = bedrock.invoke_model(
modelId="amazon.titan-embed-image-v1",
body=json.dumps({"inputImage": img_b64}),
accept="application/json",
contentType="application/json",
)
print(json.loads(resp["body"].read()))
# {"embedding": [...], "inputTextTokenCount": 0}
# AWS intentionally returns inputTextTokenCount=0 for images —
# the model is priced PER IMAGE ($0.00006/image), not per token.
# There is no "inputImageTokenCount" field — AWS does not expose one.
Relevant log output
What part of LiteLLM is this about?
SDK (litellm Python package)
What LiteLLM version are you on ?
v1.83.8
Twitter / LinkedIn details
The two patches
litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.pyindex 07b04734c3..17da5b30f7 100644@@ -88,14 +88,19 @@ class AmazonTitanMultimodalEmbeddingG1Config: prompt_tokens_details: Optional[PromptTokensDetailsWrapper] = None if image_count > 0:+ # Bedrock returns inputTextTokenCount=0 for images (billed per image),+ # so text tokens are kept separate from the image count for cost calc. prompt_tokens_details = PromptTokensDetailsWrapper(+ text_tokens=total_prompt_tokens, image_count=image_count, ) + # Count each image as one prompt unit so usage is non-zero for image inputs+ prompt_tokens = total_prompt_tokens + image_count usage = Usage(- prompt_tokens=total_prompt_tokens,+ prompt_tokens=prompt_tokens, completion_tokens=0,- total_tokens=total_prompt_tokens,+ total_tokens=prompt_tokens, prompt_tokens_details=prompt_tokens_details, ) return EmbeddingResponse(model=model, usage=usage, data=transformed_responses)tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.pyindex a38a6612f7..7d312a4b3a 100644@@ -955,3 +955,101 @@ def test_titan_image_embedding_cost_uses_per_image_rate(): assert response.usage is not None assert response.usage.prompt_tokens_details is not None assert response.usage.prompt_tokens_details.image_count == 1+++def test_titan_multimodal_embedding_image_prompt_tokens_non_zero():+ """+ Image-only Titan multimodal embeddings must report non-zero usage.prompt_tokens,+ since Bedrock returns inputTextTokenCount=0 for images.+ """+ from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+ AmazonTitanMultimodalEmbeddingG1Config,+ )++ config = AmazonTitanMultimodalEmbeddingG1Config()++ result = config._transform_response(+ response_list=[+ {"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 0},+ {"embedding": [0.4, 0.5, 0.6], "inputTextTokenCount": 0},+ ],+ model="amazon.titan-embed-image-v1",+ batch_data=[+ {"inputImage": "/9j/4AAQSkZJRg=="},+ {"inputImage": "iVBORw0KGgo="},+ ],+ )++ assert result.usage.prompt_tokens == 2+ assert result.usage.total_tokens == 2+ assert result.usage.prompt_tokens_details.image_count == 2++ # cost must stay per-image only, not inflated by the image units in prompt_tokens+ cost = litellm.completion_cost(+ completion_response=result,+ model="bedrock/amazon.titan-embed-image-v1",+ call_type="embedding",+ )+ assert cost == pytest.approx(2 * 6e-05)+++def test_titan_multimodal_embedding_mixed_text_and_image_usage():+ """Mixed text+image batch: prompt_tokens includes text tokens and images; cost charges both."""+ from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (+ AmazonTitanMultimodalEmbeddingG1Config,+ )++ config = AmazonTitanMultimodalEmbeddingG1Config()++ result = config._transform_response(+ response_list=[+ {"embedding": [0.1, 0.2, 0.3], "inputTextTokenCount": 5},+ {"embedding": [0.4, 0.5, 0.6], "inputTextTokenCount": 0},+ ],+ model="amazon.titan-embed-image-v1",+ batch_data=[+ {"inputText": "hello world"},+ {"inputImage": "/9j/4AAQSkZJRg=="},+ ],+ )++ assert result.usage.prompt_tokens == 6+ assert result.usage.total_tokens == 6+ assert result.usage.prompt_tokens_details.text_tokens == 5+ assert result.usage.prompt_tokens_details.image_count == 1++ cost = litellm.completion_cost(+ completion_response=result,+ model="bedrock/amazon.titan-embed-image-v1",+ call_type="embedding",+ )+ assert cost == pytest.approx(5 * 8e-07 + 6e-05)+++def test_titan_image_embedding_e2e_prompt_tokens_non_zero():+ """End-to-end: litellm.embedding() with an image data URI returns prompt_tokens > 0."""+ client = HTTPHandler()++ with patch.object(client, "post") as mock_post:+ mock_response = Mock()+ mock_response.status_code = 200+ mock_response.text = json.dumps(+ {"embedding": [0.1] * 1024, "inputTextTokenCount": 0}+ )+ mock_response.json = lambda: json.loads(mock_response.text)+ mock_post.return_value = mock_response++ response = litellm.embedding(+ model="bedrock/amazon.titan-embed-image-v1",+ input=["data:image/png;base64,iVBORw0KGgoAAAANSUhEUg=="],+ client=client,+ aws_access_key_id="fake",+ aws_secret_access_key="fake",+ aws_region_name="us-east-1",+ )++ assert response.usage.prompt_tokens == 1+ assert response.usage.total_tokens == 1+ assert litellm.completion_cost(completion_response=response) == pytest.approx(+ 6e-05+ )
litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.pyindex 07b04734c30..d67f936aa88 100644@@ -93,9 +93,9 @@ class AmazonTitanMultimodalEmbeddingG1Config: ) usage = Usage(- prompt_tokens=total_prompt_tokens,+ prompt_tokens=total_prompt_tokens + image_count, completion_tokens=0,- total_tokens=total_prompt_tokens,+ total_tokens=total_prompt_tokens + image_count, prompt_tokens_details=prompt_tokens_details, ) return EmbeddingResponse(model=model, usage=usage, data=transformed_responses)tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.pyindex a38a6612f79..d2813ecb3a8 100644@@ -863,6 +863,8 @@ def test_titan_multimodal_embedding_image_cost_tracking(): ) assert result.usage is not None+ assert result.usage.prompt_tokens == 1 # image_count included in prompt_tokens+ assert result.usage.total_tokens == 1 assert result.usage.prompt_tokens_details is not None assert result.usage.prompt_tokens_details.image_count == 1