Backtestspydantic-airun-20260918-215936#6305

Issue #6305 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

ResponseReasoningTextDeltaEvent handler drops reasoning content — ThinkingPart.content stays empty for raw CoT providers (gpt-oss, vLLM, LM Studio, litellm)

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches fix the two paths named in the issue: they add `content=chunk.delta` to the streaming `ResponseReasoningTextDeltaEvent` handler (identical fix), and they populate `ThinkingPart.content` from `raw_content` in the non-streaming path. The decisive differences are consistency and the multiturn round-trip. (1) Join semantics: the streaming path builds content by concatenating deltas with NO separator (`handle_thinking_delta(content=chunk.delta)`), so Patch B's `''.join(raw_content)` matches streaming behavior, while Patch A's `' '.join(raw_content)` makes the non-streaming result inconsistent with its own streaming path — the same reasoning text would render differently depending on transport. (2) Round-trip regression: once `ThinkingPart.content` is non-empty, the send-back logic at ~line 2976 (`if item.content: reasoning_item['summary'] = ...`) newly fires, so the raw CoT gets sent BOTH as a summary and via the existing raw_content reconstruction — duplicated reasoning on subsequent turns. Patch A never touches this path and thus reintroduces duplication in exactly the multiturn scenario the issue calls out (`test_openai_responses_raw_cot_sent_in_multiturn`). Patch B guards it precisely (`if item.content and item.content != ''.join(raw_content or [])`), and its choice of `''.join` in the parse path is what makes this equality check reliable — the two edits are coherent. Both update the same tests to fail-before/pass-after. Patch B's extra edit is a necessary correctness fix, not scope creep. The guard is slightly subtle but justified. Patch B is the more complete and internally consistent fix.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

10m 59s3 turns

Wall-clock time in the container

Full test suite

No new failures

7,852 passed · 530 skipped · 8,417 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

15vs 22 merged

Changed lines, ours vs merged

Initial Checks

Description

When using OpenAIResponsesModel with providers that emit response.reasoning_text.delta events (instead of response.reasoning_summary_text.delta), the reasoning content is dropped — ThinkingPart.content remains empty (or "").

This affects providers like:

  • gpt-oss via LM Studio / OpenRouter
  • vLLM (both native /v1/responses and via litellm bridge)
  • Any OpenAI-compatible provider that uses reasoning_text content type instead of reasoning_summary_text

The official OpenAI Responses API only emits response.reasoning_summary_text.delta, but third-party implementations use response.reasoning_text.delta for raw chain-of-thought. The ResponseReasoningTextDeltaEvent handler was added in response to #3191, but it is incomplete — it does not pass content=chunk.delta to handle_thinking_delta().

Two code paths are affected:

  1. Streaming (openai.py ~line 3704): ResponseReasoningTextDeltaEvent handler calls handle_thinking_delta() without content=chunk.delta. The delta text only goes to provider_details via _make_raw_content_updater. Compare with the ResponseReasoningSummaryTextDeltaEvent handler (~line 3690) which correctly passes content=chunk.delta.
  1. Non-streaming (openai.py ~line 2004): When item.summary is empty but item.content has reasoning_text entries, ThinkingPart(content='') is created. The raw content is only stored in provider_details['raw_content'].

Expected: ThinkingPart.content should contain the actual reasoning text, not an empty string.

Minimal, Reproducible Example

# Simulated streaming response with reasoning_text.delta events
# (as emitted by vLLM, LM Studio with gpt-oss, or litellm bridge)

from pydantic_ai.models.openai import OpenAIResponsesModel, OpenAIResponsesStreamedResponse

# When a provider sends:
#   {"type": "response.reasoning_text.delta", "item_id": "rs_123", "delta": "Let me think...", "content_index": 0}
#   {"type": "response.reasoning_text.delta", "item_id": "rs_123", "delta": "The answer is 4.", "content_index": 0}
#
# The handler processes the event but NEVER passes chunk.delta as content:
#
#   elif isinstance(chunk, responses.ResponseReasoningTextDeltaEvent):
#       for event in self._parts_manager.handle_thinking_delta(
#           vendor_part_id=chunk.item_id,
#           # MISSING: content=chunk.delta,
#           id=chunk.item_id,
#           ...
#       ):
#
# Result: ThinkingPart(content='') instead of ThinkingPart(content='Let me think... The answer is 4.')

You can verify by running the existing tests test_openai_responses_raw_cot_only, test_openai_responses_raw_cot_stream_openrouter, and test_openai_responses_raw_cot_sent_in_multiturn — all assert content='' for ThinkingPart when raw CoT is present, which is the bug.

Logfire Trace

_No Logfire trace available — issue found via curl testing against vLLM + litellm proxy._

Python, Pydantic AI & LLM client version

  • Python: 3.13
  • Pydantic AI: 2.5.1
  • LLM provider SDK: openai>=1.99.0 (via vLLM v0.x + litellm proxy)

The two patches

Our patch+8 −7
pydantic_ai_slim/pydantic_ai/models/openai.pyindex 7cd80947..ccf0213b 100644@@ -2004,7 +2004,7 @@ class OpenAIResponsesModel(Model[AsyncOpenAI]):                 elif signature or provider_details:                     items.append(                         ThinkingPart(-                            content='',+                            content=''.join(raw_content) if raw_content else '',                             id=item.id,                             signature=signature,                             provider_name=self.system,@@ -2976,7 +2976,7 @@ class OpenAIResponsesModel(Model[AsyncOpenAI]):                                 )                                 openai_messages.append(reasoning_item) -                            if item.content:+                            if item.content and item.content != ''.join(raw_content or []):                                 # The check above guarantees that `reasoning_item` is not None                                 assert reasoning_item is not None                                 reasoning_item['summary'] = [@@ -3702,6 +3702,7 @@ class OpenAIResponsesStreamedResponse(StreamedResponse):                     # Handle raw CoT from gpt-oss models using callback pattern                     for event in self._parts_manager.handle_thinking_delta(                         vendor_part_id=chunk.item_id,+                        content=chunk.delta,                         id=chunk.item_id,                         provider_name=self.provider_name,                         provider_details=_make_raw_content_updater(chunk.delta, chunk.content_index),tests/models/test_openai_responses.pyindex dfeb4396..5ae71c17 100644@@ -9957,7 +9957,7 @@ async def test_openai_responses_raw_cot_only(allow_model_requests: None):             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Let me think about this...The answer is 4.',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Let me think about this...', 'The answer is 4.']},@@ -10148,7 +10148,7 @@ async def test_openai_responses_raw_cot_stream_openrouter(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='The user asks: "What is 2+2?" They expect a straightforward answer: 4. Just answer 4.',                         id='rs_tmp_2kbe7x16sax',                         provider_name='openrouter',                         provider_details={@@ -10294,7 +10294,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Raw CoT step 1Raw CoT step 2',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10331,7 +10331,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Raw CoT step 1Raw CoT step 2',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10400,7 +10400,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Raw CoT step 1Raw CoT step 2',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']}, 
Merged fix · PR #+15 −7
pydantic_ai_slim/pydantic_ai/models/openai.pyindex 7cd8094790..9f38a15fb4 100644@@ -2002,9 +2002,14 @@ class OpenAIResponsesModel(Model[AsyncOpenAI]):                         signature = None                         provider_details = {}                 elif signature or provider_details:+                    # When there are no summaries but there is raw CoT content+                    # (e.g. from gpt-oss models, vLLM, or litellm bridge), use+                    # the raw content text as the ThinkingPart content so it is+                    # visible to consumers, not just stored in provider_details.+                    raw_content_text = ' '.join(raw_content) if raw_content else ''                     items.append(                         ThinkingPart(-                            content='',+                            content=raw_content_text,                             id=item.id,                             signature=signature,                             provider_name=self.system,@@ -3699,9 +3704,12 @@ class OpenAIResponsesStreamedResponse(StreamedResponse):                         yield event                  elif isinstance(chunk, responses.ResponseReasoningTextDeltaEvent):-                    # Handle raw CoT from gpt-oss models using callback pattern+                    # Handle raw CoT from gpt-oss models and OpenAI-compatible providers+                    # (e.g. vLLM, LM Studio, litellm bridge) that use `reasoning_text` events+                    # instead of `reasoning_summary_text` events.                     for event in self._parts_manager.handle_thinking_delta(                         vendor_part_id=chunk.item_id,+                        content=chunk.delta,                         id=chunk.item_id,                         provider_name=self.provider_name,                         provider_details=_make_raw_content_updater(chunk.delta, chunk.content_index),tests/models/test_openai_responses.pyindex dfeb439621..1c9dc96fea 100644@@ -9957,7 +9957,7 @@ async def test_openai_responses_raw_cot_only(allow_model_requests: None):             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Let me think about this... The answer is 4.',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Let me think about this...', 'The answer is 4.']},@@ -10148,7 +10148,7 @@ async def test_openai_responses_raw_cot_stream_openrouter(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='The user asks: "What is 2+2?" They expect a straightforward answer: 4. Just answer 4.',                         id='rs_tmp_2kbe7x16sax',                         provider_name='openrouter',                         provider_details={@@ -10294,7 +10294,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Raw CoT step 1 Raw CoT step 2',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10331,7 +10331,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Raw CoT step 1 Raw CoT step 2',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10400,7 +10400,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests:             ModelResponse(                 parts=[                     ThinkingPart(-                        content='',+                        content='Raw CoT step 1 Raw CoT step 2',                         id='rs_123',                         provider_name='openai',                         provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']}, 

What the agent reported

I'm waiting for the fail-before-fix test run (monitored) to report. Standing by.