Backtestslitellmrun-20261019-033420#42869

Issue #42869 Merged fix PR #43414 Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: streaming /v1/responses emits no reasoning item for signature-only thinking (Claude Fable 5.1 / Opus 5.5 default, Bedrock adaptive), so reasoning cannot be replayed

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

The production changes are functionally equivalent: both relax `_flush_thinking_block` to keep any signed block regardless of thinking text, and both open the reasoning item in `_ensure_output_item_for_chunk` when the delta carries a thinking block with a `signature` or redacted `data`. The independent tests confirm both fix the issue (all 5 base failures pass on A and B). The differences are in tests and scope. Patch A's streaming test is parametrized over sync/async and over both signature-only and redacted_thinking blocks, asserts the exact `[reasoning, message]` ordering for both the added events and `response.completed`, and checks that `encrypted_content` round-trips to the original block; it also pins the chunk-builder behaviour. Patch B's streaming test covers only signature-only thinking and asserts weaker properties (first item is reasoning, one reasoning item, signature matches), leaving its redacted-thinking (`data`) branch untested. More importantly, Patch B deletes an unrelated 93-line Vertex Gemma test file under `tests/test_litellm/...` and re-adds those tests into `tests/unit/llms/vertex_ai/...`, relying on a `_make_gemma_vertex_response` helper whose existence in that file is not visible in the diff. This is unrelated churn that has nothing to do with the issue, obscures the fix, and introduces risk of a broken test module. Patch A also documents the helper's intent with a docstring explaining why textless blocks still open the reasoning item. A is the cleaner, better-tested, correctly-scoped fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

27m 20s

Wall-clock time in the container

Full test suite

No new failures

64 passed · 0 skipped · 64 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

89vs 245 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and this is not a duplicate (no hits for "signature-only thinking" with streaming /v1/responses, _ensure_output_item_for_chunk, or a missing reasoning item with encrypted_content).

What happened?

On streaming /v1/responses backed by an Anthropic model (the chat -> Responses bridge, litellm/responses/litellm_completion_transformation/streaming_iterator.py), a thinking block that has a signature but empty thinking text never becomes a reasoning output item. No response.output_item.added for reasoning is streamed, and response.completed has no reasoning item, so there is no encrypted_content to replay on the next turn. The reasoning tokens are still billed (usage.output_tokens_details.reasoning_tokens is non-zero)

That shape is common: Claude Fable 5.1 and Claude Opus 5.5 return {"type": "thinking", "thinking": "", "signature": "..."} by default, and Bedrock adaptive thinking returns it with or without a reasoning effort. Non-streaming /v1/responses is not affected

Two places key on the thinking text:

  • LiteLLMCompletionStreamingIterator._ensure_output_item_for_chunk opens the reasoning item only when delta.reasoning_content is non-empty. A signature-only block arrives as delta.thinking_blocks=[{"type": "thinking", "thinking": "", "signature": "..."}] with no reasoning_content, so the first output item opened is the message
  • ChunkProcessor.get_combined_thinking_content (_flush_thinking_block) keeps an assembled block only when it collected thinking text, so the signed block is also missing from the assembled response that response.completed is built from

Measured through the proxy against real providers, streaming /v1/responses, is a reasoning item emitted?

backendno reasoning effortreasoning.effort: "high"
Bedrock (Claude Opus 4.8, Fable 5.1, Opus 5.5)nono
Vertex AI (Claude Fable 5.1, Opus 5.5)noyes
Vertex AI, deployment sets thinking: {"type": "adaptive", ...}nono

Reasoning tokens were billed in every cell. The "yes" cell is the only one where the provider returns thinking text, which is why the bug can look provider-specific

User Flow

Before a (hypothetical) fix: a client using streaming Responses with a Claude model loses its reasoning between turns

  1. They send POST https://litellm-domain/v1/responses with "stream": true and "include": ["reasoning.encrypted_content"] to a bedrock/us.anthropic.claude-opus-4-8 (or Fable 5.1 / Opus 5.5) deployment
  2. The stream opens only a message item, and response.completed.response.output is [message], while usage.output_tokens_details.reasoning_tokens is non-zero
  3. They replay output on the next turn. There is no reasoning item to replay, so the model continues without its earlier reasoning

After a (hypothetical) fix: the same request streams reasoning then message, and response.completed.response.output carries the reasoning item with encrypted_content, ready to replay. Responses whose thinking arrives as text are unchanged

Proof the bug occurs

A litellm --config proxy on main @ a3d791f348, one deployment pointed at a stand-in upstream that streams an Anthropic message with one thinking block, then text:

model_list:
  - model_name: anthropic-fable
    litellm_params:
      model: anthropic/claude-fable-5-1
      api_base: http://127.0.0.1:9913
      api_key: fake
curl -sN http://localhost:4000/v1/responses \
  -H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
  -d '{"model":"anthropic-fable","stream":true,"input":"What is 2+2?","max_output_tokens":200,
       "include":["reasoning.encrypted_content"]}'
upstream thinking blockoutput_item.added streamedresponse.completed outputreasoning encrypted_content
thinking: "", signature set (default Fable 5.1 shape)message[message]missing
thinking: "Two plus two is four.", signature setreasoning, message[reasoning, message]set

The same two unit-level shapes are pinned in the linked PR's test, which fails on unpatched main

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.101.0-rc.1 (present on main @ a3d791f348)

Twitter / LinkedIn details

_No response_

The two patches

Our patch+87 −2
litellm/litellm_core_utils/streaming_chunk_builder_utils.pyindex d975c3551f..953ea2ecd1 100644@@ -685,7 +685,9 @@ class ChunkProcessor:          def _flush_thinking_block() -> None:             nonlocal current_thinking_text_parts, current_signature-            if len(current_thinking_text_parts) > 0 and current_signature:+            # A signed block is replayable even when its thinking text is empty+            # (signature-only thinking, e.g. Bedrock adaptive thinking).+            if current_signature:                 thinking_blocks.append(                     ChatCompletionThinkingBlock(                         type="thinking",litellm/responses/litellm_completion_transformation/streaming_iterator.pyindex 5173cd04a8..44c1394496 100644@@ -924,6 +924,20 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):             else:                 raise StopAsyncIteration +    @staticmethod+    def _delta_starts_reasoning(delta: object) -> bool:+        """A delta opens the reasoning item when it carries reasoning text or a replayable thinking block.++        Signature-only thinking blocks (``thinking == ""``) and redacted thinking carry no text but+        still become the reasoning item's ``encrypted_content`` in ``response.completed``.+        """+        if getattr(delta, "reasoning_content", None):+            return True+        thinking_blocks: Final = getattr(delta, "thinking_blocks", None) or ()+        return any(+            isinstance(block, dict) and bool(block.get("signature") or block.get("data")) for block in thinking_blocks+        )+     def _ensure_output_item_for_chunk(self, chunk: ModelResponseStream) -> None:         # Change: Never return a value, just enqueue output item events         if self.sent_output_item_added_event:@@ -936,7 +950,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):         self.sent_output_item_added_event = True          # Reasoning-first-        if hasattr(delta, "reasoning_content") and delta.reasoning_content:+        if self._delta_starts_reasoning(delta):             self._reasoning_active = True             if self._cached_reasoning_item_id is None:                 self._cached_reasoning_item_id = f"rs_{uuid.uuid4()}"tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.pyindex af763da2d8..e99406acf3 100644@@ -1795,3 +1795,23 @@ def test_calculate_usage_keeps_a_reported_count_over_a_later_chunks_zero() -> No     )      assert (usage.prompt_tokens, usage.completion_tokens, usage.total_tokens) == (5, 17, 22)+++def test_get_combined_thinking_content_keeps_signature_only_block():+    chunk = ModelResponseStream(+        id="chatcmpl-123",+        object="chat.completion.chunk",+        created=1234567890,+        model="claude-fable-5-1",+        choices=[+            StreamingChoices(+                index=0,+                delta=Delta(thinking_blocks=[{"type": "thinking", "thinking": "", "signature": "sig_fable_default"}]),+                finish_reason=None,+            )+        ],+    )++    result = ChunkProcessor(chunks=[chunk]).get_combined_thinking_content([chunk])++    assert result == [{"type": "thinking", "thinking": "", "signature": "sig_fable_default"}]tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.pyindex 8fbba0dbf8..95ce02ff94 100644@@ -1131,3 +1131,52 @@ async def test_plain_text_stream_announces_exactly_one_message_item(sync_mode: b             ResponsesAPIStreamEvents.OUTPUT_TEXT_DONE,         ):             assert event.item_id == message_item_adds[0].item.id+++def _thinking_block_chunk(thinking_block: dict) -> ModelResponseStream:+    return ModelResponseStream(+        id=CHAT_COMPLETION_ID,+        created=1748575031,+        model="claude-fable-5-1",+        object="chat.completion.chunk",+        choices=[+            StreamingChoices(+                index=0,+                delta=Delta(role="assistant", thinking_blocks=[thinking_block]),+                finish_reason=None,+            )+        ],+    )+++@pytest.mark.parametrize("sync_mode", [True, False])+@pytest.mark.parametrize(+    "thinking_block",+    [+        {"type": "thinking", "thinking": "", "signature": "sig_fable_default"},+        {"type": "redacted_thinking", "data": "EuoBCoYBGAIi_redacted"},+    ],+    ids=["signature_only_thinking", "redacted_thinking"],+)+@pytest.mark.asyncio+async def test_textless_thinking_block_streams_replayable_reasoning_item(sync_mode: bool, thinking_block: dict):+    iterator: Final = _build_iterator(+        [+            _thinking_block_chunk(thinking_block),+            _chunk("4"),+            _chunk("", finish_reason="stop"),+        ]+    )++    events: Final = await _collect_events(iterator, sync_mode)++    added_item_types: Final = [+        event.item.type for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED+    ]+    assert added_item_types == ["reasoning", "message"], added_item_types+    completed: Final = next(+        event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED+    )+    output: Final = completed.response.output+    assert [item.type for item in output] == ["reasoning", "message"], output+    assert json.loads(output[0].encrypted_content) == [thinking_block] 
Merged fix · PR #43414+150 −95
litellm/litellm_core_utils/streaming_chunk_builder_utils.pyindex d975c3551f3..67684a230e3 100644@@ -685,7 +685,7 @@ class ChunkProcessor:          def _flush_thinking_block() -> None:             nonlocal current_thinking_text_parts, current_signature-            if len(current_thinking_text_parts) > 0 and current_signature:+            if current_signature:                 thinking_blocks.append(                     ChatCompletionThinkingBlock(                         type="thinking",litellm/responses/litellm_completion_transformation/streaming_iterator.pyindex 5173cd04a89..21a33c17ab8 100644@@ -73,6 +73,11 @@ def _output_items_with_id(items: tuple[Any, ...], item_type: str, item_id: str |     )  +def _delta_has_signed_thinking_block(delta: object) -> bool:+    blocks: Final = getattr(delta, "thinking_blocks", None) or ()+    return any(isinstance(b, dict) and (b.get("signature") or b.get("data")) for b in blocks)++ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):     """     Async iterator for processing streaming responses from the Responses API.@@ -936,7 +941,7 @@ class LiteLLMCompletionStreamingIterator(ResponsesAPIStreamingIterator):         self.sent_output_item_added_event = True          # Reasoning-first-        if hasattr(delta, "reasoning_content") and delta.reasoning_content:+        if (hasattr(delta, "reasoning_content") and delta.reasoning_content) or _delta_has_signed_thinking_block(delta):             self._reasoning_active = True             if self._cached_reasoning_item_id is None:                 self._cached_reasoning_item_id = f"rs_{uuid.uuid4()}"tests/test_litellm/llms/vertex_ai/vertex_gemma_models/test_vertex_gemma_transformation.pydeleted file mode 100644index 294d26b2e58..00000000000@@ -1,93 +0,0 @@-import json-from collections.abc import AsyncIterator-from types import SimpleNamespace-from typing import Any, cast--import httpx-import pytest--import litellm-from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper-from litellm.main import vertex_gemma_chat_completion-from litellm.types.llms.openai import OutputTextDeltaEvent, ResponseCompletedEvent, ResponsesAPIStreamingResponse--_VERTEX_URL = "https://example.invalid/v1/projects/test/locations/us-central1/endpoints/test:predict"-_MESSAGES = [{"role": "user", "content": "Reply exactly READY"}]-_FAKE_CREDENTIALS = "gemma-test-credentials"---def _vertex_response():-    return {-        "predictions": {-            "id": "chatcmpl-stream-test",-            "created": 1759863903,-            "model": "google/gemma-3-12b-it",-            "object": "chat.completion",-            "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "READY"}}],-            "usage": {"prompt_tokens": 14, "completion_tokens": 1, "total_tokens": 15},-        }-    }---@pytest.fixture(autouse=True)-def _cached_access_token():-    """Serve a fake token from the handler's credential cache so no auth round-trip runs."""-    cache = vertex_gemma_chat_completion._credentials_project_mapping-    key = (_FAKE_CREDENTIALS, "test")-    cache[key] = (SimpleNamespace(token="fake-token", expired=False), "test")-    yield-    cache.pop(key, None)---def test_sync_gemma_stream():-    captured: dict[str, Any] = {}--    def handle(request: httpx.Request) -> httpx.Response:-        captured["body"] = json.loads(request.content)-        return httpx.Response(200, json=_vertex_response())--    stream = litellm.completion(-        model="vertex_ai/gemma/test-model",-        messages=_MESSAGES,-        stream=True,-        api_base=_VERTEX_URL,-        vertex_project="test",-        vertex_location="us-central1",-        vertex_credentials=_FAKE_CREDENTIALS,-        client=httpx.Client(transport=httpx.MockTransport(handle)),-    )--    assert isinstance(stream, CustomStreamWrapper)-    chunks = list(stream)--    assert "stream" not in captured["body"]["instances"][0]-    assert len(chunks) == 2-    assert chunks[0].choices[0].delta.content == "READY"-    assert chunks[1].choices[0].finish_reason == "stop"---@pytest.mark.asyncio-async def test_async_gemma_responses_stream():-    captured: dict[str, Any] = {}--    def handle(request: httpx.Request) -> httpx.Response:-        captured["body"] = json.loads(request.content)-        return httpx.Response(200, json=_vertex_response())--    response = await litellm.aresponses(-        model="vertex_ai/gemma/test-model",-        input="Reply exactly READY",-        stream=True,-        api_base=_VERTEX_URL,-        vertex_project="test",-        vertex_location="us-central1",-        vertex_credentials=_FAKE_CREDENTIALS,-        client=httpx.AsyncClient(transport=httpx.MockTransport(handle)),-    )-    events = [event async for event in cast(AsyncIterator[ResponsesAPIStreamingResponse], response)]--    assert "stream" not in captured["body"]["instances"][0]-    assert "READY" in "".join(event.delta for event in events if isinstance(event, OutputTextDeltaEvent))-    assert isinstance(events[-1], ResponseCompletedEvent)-    assert events[-1].response.usage is not None-    assert events[-1].response.usage.total_tokens == 15tests/unit/litellm_core_utils/test_streaming_chunk_builder_utils.pyindex af763da2d87..aaf877df364 100644@@ -236,6 +236,31 @@ def test_get_combined_thinking_content_preserves_interleaved_blocks():     assert result[2]["signature"] == "sig_block2"  +def test_get_combined_thinking_content_keeps_signed_block_without_thinking_text():+    chunks: Final = [+        ModelResponseStream(+            id="chatcmpl-123",+            object="chat.completion.chunk",+            created=1234567890,+            model="claude-sonnet-4-20250514",+            choices=[+                StreamingChoices(+                    index=0,+                    delta=Delta(thinking_blocks=[{"type": "thinking", "thinking": "", "signature": "sig_only"}]),+                    finish_reason=None,+                )+            ],+        )+    ]++    result: Final = ChunkProcessor(chunks=chunks).get_combined_thinking_content(chunks)++    assert result is not None+    assert [(block["type"], block["thinking"], block["signature"]) for block in result] == [+        ("thinking", "", "sig_only")+    ]++ def test_cache_read_input_tokens_retained():     chunk1 = ModelResponseStream(         id="chatcmpl-95aabb85-c39f-443d-ae96-0370c404d70c",tests/unit/llms/vertex_ai/vertex_gemma_models/test_vertex_gemma_transformation.pyindex 97f4f290958..92e684e42c8 100644@@ -1303,3 +1303,81 @@ class TestVertexGemmaCompletion:         mock_async_post.assert_awaited_once()         assert mock_async_post.call_args.kwargs["client"] is None         assert response.choices[0].message.content == "default async handler fallback"+++_GEMMA_VERTEX_URL = "https://example.invalid/v1/projects/test/locations/us-central1/endpoints/test:predict"+_FAKE_GEMMA_CREDENTIALS = "gemma-test-credentials"+++@pytest.fixture+def _gemma_cached_access_token():+    """Serve a fake token from the handler's credential cache so no auth round-trip runs."""+    from types import SimpleNamespace++    from litellm.main import vertex_gemma_chat_completion++    cache = vertex_gemma_chat_completion._credentials_project_mapping+    key = (_FAKE_GEMMA_CREDENTIALS, "test")+    cache[key] = (SimpleNamespace(token="fake-token", expired=False), "test")+    yield+    cache.pop(key, None)+++def test_sync_gemma_stream(_gemma_cached_access_token):+    import httpx++    from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper++    captured = {}++    def handle(request):+        captured["body"] = json.loads(request.content)+        return httpx.Response(200, json=_make_gemma_vertex_response(content="READY", total_tokens=15))++    stream = litellm.completion(+        model="vertex_ai/gemma/test-model",+        messages=[{"role": "user", "content": "Reply exactly READY"}],+        stream=True,+        api_base=_GEMMA_VERTEX_URL,+        vertex_project="test",+        vertex_location="us-central1",+        vertex_credentials=_FAKE_GEMMA_CREDENTIALS,+        client=httpx.Client(transport=httpx.MockTransport(handle)),+    )++    assert isinstance(stream, CustomStreamWrapper)+    chunks = list(stream)++    assert "stream" not in captured["body"]["instances"][0]+    assert len(chunks) == 2+    assert chunks[0].choices[0].delta.content == "READY"+    assert chunks[1].choices[0].finish_reason == "stop"+++@pytest.mark.asyncio+async def test_async_gemma_responses_stream(_gemma_cached_access_token):+    import httpx++    captured = {}++    def handle(request):+        captured["body"] = json.loads(request.content)+        return httpx.Response(200, json=_make_gemma_vertex_response(content="READY", total_tokens=15))++    response = await litellm.aresponses(+        model="vertex_ai/gemma/test-model",+        input="Reply exactly READY",+        stream=True,+        api_base=_GEMMA_VERTEX_URL,+        vertex_project="test",+        vertex_location="us-central1",+        vertex_credentials=_FAKE_GEMMA_CREDENTIALS,+        client=httpx.AsyncClient(transport=httpx.MockTransport(handle)),+    )+    events = [event async for event in cast(AsyncIterator[ResponsesAPIStreamingResponse], response)]++    assert "stream" not in captured["body"]["instances"][0]+    assert "READY" in "".join(event.delta for event in events if isinstance(event, OutputTextDeltaEvent))+    assert isinstance(events[-1], ResponseCompletedEvent)+    assert events[-1].response.usage is not None+    assert events[-1].response.usage.total_tokens == 15tests/unit/responses/litellm_completion_transformation/test_streaming_iterator_transformation.pyindex 8fbba0dbf87..041bcf1b6d7 100644@@ -978,6 +978,25 @@ def _reasoning_chunk(reasoning: str, finish_reason: str | None = None) -> ModelR     )  +def _signature_only_thinking_chunk(signature: str) -> ModelResponseStream:+    return ModelResponseStream(+        id=CHAT_COMPLETION_ID,+        created=1748575031,+        model="claude-haiku-4-5",+        object="chat.completion.chunk",+        choices=[+            StreamingChoices(+                index=0,+                delta=Delta(+                    role="assistant",+                    thinking_blocks=[{"type": "thinking", "thinking": "", "signature": signature}],+                ),+                finish_reason=None,+            )+        ],+    )++ async def _collect_events(     iterator: LiteLLMCompletionStreamingIterator, sync_mode: bool ) -> list[BaseLiteLLMOpenAIResponseObject]:@@ -1015,6 +1034,27 @@ async def test_tool_only_stream_emits_no_message_item_events(sync_mode: bool):     assert any(getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED for event in events)  +@pytest.mark.parametrize("sync_mode", [True, False])+@pytest.mark.asyncio+async def test_signature_only_thinking_streams_a_replayable_reasoning_item(sync_mode: bool):+    iterator: Final = _build_iterator([_signature_only_thinking_chunk("sig_only"), _chunk("4", finish_reason="stop")])++    events: Final = await _collect_events(iterator, sync_mode)++    added_item_types: Final = [+        event.item.type+        for event in events+        if getattr(event, "type", None) == ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED+    ]+    completed: Final = next(+        event for event in events if getattr(event, "type", None) == ResponsesAPIStreamEvents.RESPONSE_COMPLETED+    )+    reasoning_items: Final = [item for item in completed.response.output if getattr(item, "type", None) == "reasoning"]+    assert added_item_types[0] == "reasoning"+    assert len(reasoning_items) == 1+    assert json.loads(reasoning_items[0].encrypted_content)[0]["signature"] == "sig_only"++ @pytest.mark.parametrize("sync_mode", [True, False]) @pytest.mark.asyncio async def test_reasoning_then_text_announces_message_item_before_text_events(sync_mode: bool):