Backtestslitellmrun-20261019-033420#42869
[Bug]: streaming /v1/responses emits no reasoning item for signature-only thinking (Claude Fable 5.1 / Opus 5.5 default, Bedrock adaptive), so reasoning cannot be replayed
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The production changes are functionally equivalent: both relax `_flush_thinking_block` to keep any signed block regardless of thinking text, and both open the reasoning item in `_ensure_output_item_for_chunk` when the delta carries a thinking block with a `signature` or redacted `data`. The independent tests confirm both fix the issue (all 5 base failures pass on A and B). The differences are in tests and scope. Patch A's streaming test is parametrized over sync/async and over both signature-only and redacted_thinking blocks, asserts the exact `[reasoning, message]` ordering for both the added events and `response.completed`, and checks that `encrypted_content` round-trips to the original block; it also pins the chunk-builder behaviour. Patch B's streaming test covers only signature-only thinking and asserts weaker properties (first item is reasoning, one reasoning item, signature matches), leaving its redacted-thinking (`data`) branch untested. More importantly, Patch B deletes an unrelated 93-line Vertex Gemma test file under `tests/test_litellm/...` and re-adds those tests into `tests/unit/llms/vertex_ai/...`, relying on a `_make_gemma_vertex_response` helper whose existence in that file is not visible in the diff. This is unrelated churn that has nothing to do with the issue, obscures the fix, and introduces risk of a broken test module. Patch A also documents the helper's intent with a docstring explaining why textless blocks still open the reasoning item. A is the cleaner, better-tested, correctly-scoped fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
27m 20s
Wall-clock time in the container
Full test suite
No new failures
64 passed · 0 skipped · 64 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
89vs 245 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and this is not a duplicate (no hits for "signature-only thinking" with streaming
/v1/responses,_ensure_output_item_for_chunk, or a missing reasoning item withencrypted_content).
What happened?
On streaming /v1/responses backed by an Anthropic model (the chat -> Responses bridge, litellm/responses/litellm_completion_transformation/streaming_iterator.py), a thinking block that has a signature but empty thinking text never becomes a reasoning output item. No response.output_item.added for reasoning is streamed, and response.completed has no reasoning item, so there is no encrypted_content to replay on the next turn. The reasoning tokens are still billed (usage.output_tokens_details.reasoning_tokens is non-zero)
That shape is common: Claude Fable 5.1 and Claude Opus 5.5 return {"type": "thinking", "thinking": "", "signature": "..."} by default, and Bedrock adaptive thinking returns it with or without a reasoning effort. Non-streaming /v1/responses is not affected
Two places key on the thinking text:
LiteLLMCompletionStreamingIterator._ensure_output_item_for_chunkopens the reasoning item only whendelta.reasoning_contentis non-empty. A signature-only block arrives asdelta.thinking_blocks=[{"type": "thinking", "thinking": "", "signature": "..."}]with noreasoning_content, so the first output item opened is the messageChunkProcessor.get_combined_thinking_content(_flush_thinking_block) keeps an assembled block only when it collected thinking text, so the signed block is also missing from the assembled response thatresponse.completedis built from
Measured through the proxy against real providers, streaming /v1/responses, is a reasoning item emitted?
| backend | no reasoning effort | reasoning.effort: "high" |
|---|---|---|
| Bedrock (Claude Opus 4.8, Fable 5.1, Opus 5.5) | no | no |
| Vertex AI (Claude Fable 5.1, Opus 5.5) | no | yes |
Vertex AI, deployment sets thinking: {"type": "adaptive", ...} | no | no |
Reasoning tokens were billed in every cell. The "yes" cell is the only one where the provider returns thinking text, which is why the bug can look provider-specific
User Flow
Before a (hypothetical) fix: a client using streaming Responses with a Claude model loses its reasoning between turns
- They send
POST https://litellm-domain/v1/responseswith"stream": trueand"include": ["reasoning.encrypted_content"]to abedrock/us.anthropic.claude-opus-4-8(or Fable 5.1 / Opus 5.5) deployment - The stream opens only a
messageitem, andresponse.completed.response.outputis[message], whileusage.output_tokens_details.reasoning_tokensis non-zero - They replay
outputon the next turn. There is no reasoning item to replay, so the model continues without its earlier reasoning
After a (hypothetical) fix: the same request streams reasoning then message, and response.completed.response.output carries the reasoning item with encrypted_content, ready to replay. Responses whose thinking arrives as text are unchanged
Proof the bug occurs
A litellm --config proxy on main @ a3d791f348, one deployment pointed at a stand-in upstream that streams an Anthropic message with one thinking block, then text:
model_list:
- model_name: anthropic-fable
litellm_params:
model: anthropic/claude-fable-5-1
api_base: http://127.0.0.1:9913
api_key: fake
curl -sN http://localhost:4000/v1/responses \
-H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
-d '{"model":"anthropic-fable","stream":true,"input":"What is 2+2?","max_output_tokens":200,
"include":["reasoning.encrypted_content"]}'
| upstream thinking block | output_item.added streamed | response.completed output | reasoning encrypted_content |
|---|---|---|---|
thinking: "", signature set (default Fable 5.1 shape) | message | [message] | missing |
thinking: "Two plus two is four.", signature set | reasoning, message | [reasoning, message] | set |
The same two unit-level shapes are pinned in the linked PR's test, which fails on unpatched main
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.101.0-rc.1 (present on main @ a3d791f348)
Twitter / LinkedIn details
_No response_
The two patches
litellm/litellm_core_utils/streaming_chunk_builder_utils.pyindex d975c3551f..953ea2ecd1 100644@@ -685,7 +685,9 @@ class ChunkProcessor: def _flush_thinking_block() -> None: nonlocal current_thinking_text_parts, current_signature- if len(current_thinking_text_parts) > 0 and current_signature:+ # A signed block is replayable even when its thinking text is empty+ # (signature-only thinking, e.g. Bedrock adaptive thinking).+ if current_signature: thinking_blocks.append( ChatCompletionThinkingBlock( type="thinking",litellm/responses/litellm_completion_transformation/streaming_iterator.pyindex 5173cd04a8..44c1394496 100644@@ -924,6 +924,20 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator): else: raise StopAsyncIteration + @staticmethod+ def _delta_starts_reasoning(delta: object) -> bool:+ """A delta opens the reasoning item when it carries reasoning text or a replayable thinking block.++ Signature-only thinking blocks (``thinking == ""``) and redacted thinking carry no text but+ still become the reasoning item's ``encrypted_content`` in ``response.completed``.+ """+ if getattr(delta, "reasoning_content", None):+ return True+ thinking_blocks: Final = getattr(delta, "thinking_blocks", None) or ()+ return any(+ isinstance(block, dict) and bool(block.get("signature") or block.get("data")) for block in thinking_blocks+ )+ def _ensure_output_item_for_chunk(self, chunk: ModelResponseStream) -> None: # Change: Never return a value, just enqueue output item events if self.sent_output_item_added_event:@@ -936,7 +950,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator): self.sent_output_item_added_event = True # Reasoning-first- if hasattr(delta, "reasoning_content") and delta.reasoning_content:+ if self._delta_starts_reasoning(delta): self._reasoning_active = True if self._cached_reasoning_item_id is None: self._cached_reasoning_item_id = f"rs_{uuid.uuid4()}"tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.pyindex af763da2d8..e99406acf3 100644@@ -1795,3 +1795,23 @@ def test_calculate_usage_keeps_a_reported_count_over_a_later_chunks_zero() -> No ) assert (usage.prompt_tokens, usage.completion_tokens, usage.total_tokens) == (5, 17, 22)+++def test_get_combined_thinking_content_keeps_signature_only_block():+ chunk = ModelResponseStream(+ id="chatcmpl-123",+ object="chat.completion.chunk",+ created=1234567890,+ model="claude-fable-5-1",+ choices=[+ StreamingChoices(+ index=0,+ delta=Delta(thinking_blocks=[{"type": "thinking", "thinking": "", "signature": "sig_fable_default"}]),+ finish_reason=None,+ )+ ],+ )++ result = ChunkProcessor(chunks=[chunk]).get_combined_thinking_content([chunk])++ assert result == [{"type": "thinking", "thinking": "", "signature": "sig_fable_default"}]tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.pyindex 8fbba0dbf8..95ce02ff94 100644@@ -1131,3 +1131,52 @@ async def test_plain_text_stream_announces_exactly_one_message_item(sync_mode: b ResponsesAPIStreamEvents.OUTPUT_TEXT_DONE, ): assert event.item_id == message_item_adds[0].item.id+++def _thinking_block_chunk(thinking_block: dict) -> ModelResponseStream:+ return ModelResponseStream(+ id=CHAT_COMPLETION_ID,+ created=1748575031,+ model="claude-fable-5-1",+ object="chat.completion.chunk",+ choices=[+ StreamingChoices(+ index=0,+ delta=Delta(role="assistant", thinking_blocks=[thinking_block]),+ finish_reason=None,+ )+ ],+ )+++@pytest.mark.parametrize("sync_mode", [True, False])+@pytest.mark.parametrize(+ "thinking_block",+ [+ {"type": "thinking", "thinking": "", "signature": "sig_fable_default"},+ {"type": "redacted_thinking", "data": "EuoBCoYBGAIi_redacted"},+ ],+ ids=["signature_only_thinking", "redacted_thinking"],+)+@pytest.mark.asyncio+async def test_textless_thinking_block_streams_replayable_reasoning_item(sync_mode: bool, thinking_block: dict):+ iterator: Final = _build_iterator(+ [+ _thinking_block_chunk(thinking_block),+ _chunk("4"),+ _chunk("", finish_reason="stop"),+ ]+ )++ events: Final = await _collect_events(iterator, sync_mode)++ added_item_types: Final = [+ event.item.type for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED+ ]+ assert added_item_types == ["reasoning", "message"], added_item_types+ completed: Final = next(+ event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED+ )+ output: Final = completed.response.output+ assert [item.type for item in output] == ["reasoning", "message"], output+ assert json.loads(output[0].encrypted_content) == [thinking_block]
litellm/litellm_core_utils/streaming_chunk_builder_utils.pyindex d975c3551f3..67684a230e3 100644@@ -685,7 +685,7 @@ class ChunkProcessor: def _flush_thinking_block() -> None: nonlocal current_thinking_text_parts, current_signature- if len(current_thinking_text_parts) > 0 and current_signature:+ if current_signature: thinking_blocks.append( ChatCompletionThinkingBlock( type="thinking",litellm/responses/litellm_completion_transformation/streaming_iterator.pyindex 5173cd04a89..21a33c17ab8 100644@@ -73,6 +73,11 @@ def _output_items_with_id(items: tuple[Any, ...], item_type: str, item_id: str | ) +def _delta_has_signed_thinking_block(delta: object) -> bool:+ blocks: Final = getattr(delta, "thinking_blocks", None) or ()+ return any(isinstance(b, dict) and (b.get("signature") or b.get("data")) for b in blocks)++ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator): """ Async iterator for processing streaming responses from the Responses API.@@ -936,7 +941,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator): self.sent_output_item_added_event = True # Reasoning-first- if hasattr(delta, "reasoning_content") and delta.reasoning_content:+ if (hasattr(delta, "reasoning_content") and delta.reasoning_content) or _delta_has_signed_thinking_block(delta): self._reasoning_active = True if self._cached_reasoning_item_id is None: self._cached_reasoning_item_id = f"rs_{uuid.uuid4()}"tests/test_litellm/llms/vertex_ai/vertex_gemma_models/test_vertex_gemma_transformation.pydeleted file mode 100644index 294d26b2e58..00000000000@@ -1,93 +0,0 @@-import json-from collections.abc import AsyncIterator-from types import SimpleNamespace-from typing import Any, cast--import httpx-import pytest--import litellm-from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper-from litellm.main import vertex_gemma_chat_completion-from litellm.types.llms.openai import OutputTextDeltaEvent, ResponseCompletedEvent, ResponsesAPIStreamingResponse--_VERTEX_URL = "https://example.invalid/v1/projects/test/locations/us-central1/endpoints/test:predict"-_MESSAGES = [{"role": "user", "content": "Reply exactly READY"}]-_FAKE_CREDENTIALS = "gemma-test-credentials"---def _vertex_response():- return {- "predictions": {- "id": "chatcmpl-stream-test",- "created": 1759863903,- "model": "google/gemma-3-12b-it",- "object": "chat.completion",- "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "READY"}}],- "usage": {"prompt_tokens": 14, "completion_tokens": 1, "total_tokens": 15},- }- }---@pytest.fixture(autouse=True)-def _cached_access_token():- """Serve a fake token from the handler's credential cache so no auth round-trip runs."""- cache = vertex_gemma_chat_completion._credentials_project_mapping- key = (_FAKE_CREDENTIALS, "test")- cache[key] = (SimpleNamespace(token="fake-token", expired=False), "test")- yield- cache.pop(key, None)---def test_sync_gemma_stream():- captured: dict[str, Any] = {}-- def handle(request: httpx.Request) -> httpx.Response:- captured["body"] = json.loads(request.content)- return httpx.Response(200, json=_vertex_response())-- stream = litellm.completion(- model="vertex_ai/gemma/test-model",- messages=_MESSAGES,- stream=True,- api_base=_VERTEX_URL,- vertex_project="test",- vertex_location="us-central1",- vertex_credentials=_FAKE_CREDENTIALS,- client=httpx.Client(transport=httpx.MockTransport(handle)),- )-- assert isinstance(stream, CustomStreamWrapper)- chunks = list(stream)-- assert "stream" not in captured["body"]["instances"][0]- assert len(chunks) == 2- assert chunks[0].choices[0].delta.content == "READY"- assert chunks[1].choices[0].finish_reason == "stop"---@pytest.mark.asyncio-async def test_async_gemma_responses_stream():- captured: dict[str, Any] = {}-- def handle(request: httpx.Request) -> httpx.Response:- captured["body"] = json.loads(request.content)- return httpx.Response(200, json=_vertex_response())-- response = await litellm.aresponses(- model="vertex_ai/gemma/test-model",- input="Reply exactly READY",- stream=True,- api_base=_VERTEX_URL,- vertex_project="test",- vertex_location="us-central1",- vertex_credentials=_FAKE_CREDENTIALS,- client=httpx.AsyncClient(transport=httpx.MockTransport(handle)),- )- events = [event async for event in cast(AsyncIterator[ResponsesAPIStreamingResponse], response)]-- assert "stream" not in captured["body"]["instances"][0]- assert "READY" in "".join(event.delta for event in events if isinstance(event, OutputTextDeltaEvent))- assert isinstance(events[-1], ResponseCompletedEvent)- assert events[-1].response.usage is not None- assert events[-1].response.usage.total_tokens == 15tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.pyindex af763da2d87..aaf877df364 100644@@ -236,6 +236,31 @@ def test_get_combined_thinking_content_preserves_interleaved_blocks(): assert result[2]["signature"] == "sig_block2" +def test_get_combined_thinking_content_keeps_signed_block_without_thinking_text():+ chunks: Final = [+ ModelResponseStream(+ id="chatcmpl-123",+ object="chat.completion.chunk",+ created=1234567890,+ model="claude-sonnet-4-20250514",+ choices=[+ StreamingChoices(+ index=0,+ delta=Delta(thinking_blocks=[{"type": "thinking", "thinking": "", "signature": "sig_only"}]),+ finish_reason=None,+ )+ ],+ )+ ]++ result: Final = ChunkProcessor(chunks=chunks).get_combined_thinking_content(chunks)++ assert result is not None+ assert [(block["type"], block["thinking"], block["signature"]) for block in result] == [+ ("thinking", "", "sig_only")+ ]++ def test_cache_read_input_tokens_retained(): chunk1 = ModelResponseStream( id="chatcmpl-95aabb85-c39f-443d-ae96-0370c404d70c",tests/unit/llms/vertex_ai/vertex_gemma_models/test_vertex_gemma_transformation.pyindex 97f4f290958..92e684e42c8 100644@@ -1303,3 +1303,81 @@ class TestVertexGemmaCompletion: mock_async_post.assert_awaited_once() assert mock_async_post.call_args.kwargs["client"] is None assert response.choices[0].message.content == "default async handler fallback"+++_GEMMA_VERTEX_URL = "https://example.invalid/v1/projects/test/locations/us-central1/endpoints/test:predict"+_FAKE_GEMMA_CREDENTIALS = "gemma-test-credentials"+++@pytest.fixture+def _gemma_cached_access_token():+ """Serve a fake token from the handler's credential cache so no auth round-trip runs."""+ from types import SimpleNamespace++ from litellm.main import vertex_gemma_chat_completion++ cache = vertex_gemma_chat_completion._credentials_project_mapping+ key = (_FAKE_GEMMA_CREDENTIALS, "test")+ cache[key] = (SimpleNamespace(token="fake-token", expired=False), "test")+ yield+ cache.pop(key, None)+++def test_sync_gemma_stream(_gemma_cached_access_token):+ import httpx++ from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper++ captured = {}++ def handle(request):+ captured["body"] = json.loads(request.content)+ return httpx.Response(200, json=_make_gemma_vertex_response(content="READY", total_tokens=15))++ stream = litellm.completion(+ model="vertex_ai/gemma/test-model",+ messages=[{"role": "user", "content": "Reply exactly READY"}],+ stream=True,+ api_base=_GEMMA_VERTEX_URL,+ vertex_project="test",+ vertex_location="us-central1",+ vertex_credentials=_FAKE_GEMMA_CREDENTIALS,+ client=httpx.Client(transport=httpx.MockTransport(handle)),+ )++ assert isinstance(stream, CustomStreamWrapper)+ chunks = list(stream)++ assert "stream" not in captured["body"]["instances"][0]+ assert len(chunks) == 2+ assert chunks[0].choices[0].delta.content == "READY"+ assert chunks[1].choices[0].finish_reason == "stop"+++@pytest.mark.asyncio+async def test_async_gemma_responses_stream(_gemma_cached_access_token):+ import httpx++ captured = {}++ def handle(request):+ captured["body"] = json.loads(request.content)+ return httpx.Response(200, json=_make_gemma_vertex_response(content="READY", total_tokens=15))++ response = await litellm.aresponses(+ model="vertex_ai/gemma/test-model",+ input="Reply exactly READY",+ stream=True,+ api_base=_GEMMA_VERTEX_URL,+ vertex_project="test",+ vertex_location="us-central1",+ vertex_credentials=_FAKE_GEMMA_CREDENTIALS,+ client=httpx.AsyncClient(transport=httpx.MockTransport(handle)),+ )+ events = [event async for event in cast(AsyncIterator[ResponsesAPIStreamingResponse], response)]++ assert "stream" not in captured["body"]["instances"][0]+ assert "READY" in "".join(event.delta for event in events if isinstance(event, OutputTextDeltaEvent))+ assert isinstance(events[-1], ResponseCompletedEvent)+ assert events[-1].response.usage is not None+ assert events[-1].response.usage.total_tokens == 15tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.pyindex 8fbba0dbf87..041bcf1b6d7 100644@@ -978,6 +978,25 @@ def _reasoning_chunk(reasoning: str, finish_reason: str | None = None) -> ModelR ) +def _signature_only_thinking_chunk(signature: str) -> ModelResponseStream:+ return ModelResponseStream(+ id=CHAT_COMPLETION_ID,+ created=1748575031,+ model="claude-haiku-4-5",+ object="chat.completion.chunk",+ choices=[+ StreamingChoices(+ index=0,+ delta=Delta(+ role="assistant",+ thinking_blocks=[{"type": "thinking", "thinking": "", "signature": signature}],+ ),+ finish_reason=None,+ )+ ],+ )++ async def _collect_events( iterator: LiteLLMCompletionStreamingIterator, sync_mode: bool ) -> list[BaseLiteLLMOpenAIResponseObject]:@@ -1015,6 +1034,27 @@ async def test_tool_only_stream_emits_no_message_item_events(sync_mode: bool): assert any(getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED for event in events) +@pytest.mark.parametrize("sync_mode", [True, False])+@pytest.mark.asyncio+async def test_signature_only_thinking_streams_a_replayable_reasoning_item(sync_mode: bool):+ iterator: Final = _build_iterator([_signature_only_thinking_chunk("sig_only"), _chunk("4", finish_reason="stop")])++ events: Final = await _collect_events(iterator, sync_mode)++ added_item_types: Final = [+ event.item.type+ for event in events+ if getattr(event, "type", None) == ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED+ ]+ completed: Final = next(+ event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED+ )+ reasoning_items: Final = [item for item in completed.response.output if getattr(item, "type", None) == "reasoning"]+ assert added_item_types[0] == "reasoning"+ assert len(reasoning_items) == 1+ assert json.loads(reasoning_items[0].encrypted_content)[0]["signature"] == "sig_only"++ @pytest.mark.parametrize("sync_mode", [True, False]) @pytest.mark.asyncio async def test_reasoning_then_text_announces_message_item_before_text_events(sync_mode: bool):