Backtestspydantic-airun-20260918-215936#6305
ResponseReasoningTextDeltaEvent handler drops reasoning content — ThinkingPart.content stays empty for raw CoT providers (gpt-oss, vLLM, LM Studio, litellm)
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the two paths named in the issue: they add `content=chunk.delta` to the streaming `ResponseReasoningTextDeltaEvent` handler (identical fix), and they populate `ThinkingPart.content` from `raw_content` in the non-streaming path. The decisive differences are consistency and the multiturn round-trip. (1) Join semantics: the streaming path builds content by concatenating deltas with NO separator (`handle_thinking_delta(content=chunk.delta)`), so Patch B's `''.join(raw_content)` matches streaming behavior, while Patch A's `' '.join(raw_content)` makes the non-streaming result inconsistent with its own streaming path — the same reasoning text would render differently depending on transport. (2) Round-trip regression: once `ThinkingPart.content` is non-empty, the send-back logic at ~line 2976 (`if item.content: reasoning_item['summary'] = ...`) newly fires, so the raw CoT gets sent BOTH as a summary and via the existing raw_content reconstruction — duplicated reasoning on subsequent turns. Patch A never touches this path and thus reintroduces duplication in exactly the multiturn scenario the issue calls out (`test_openai_responses_raw_cot_sent_in_multiturn`). Patch B guards it precisely (`if item.content and item.content != ''.join(raw_content or [])`), and its choice of `''.join` in the parse path is what makes this equality check reliable — the two edits are coherent. Both update the same tests to fail-before/pass-after. Patch B's extra edit is a necessary correctness fix, not scope creep. The guard is slightly subtle but justified. Patch B is the more complete and internally consistent fix.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
10m 59s3 turns
Wall-clock time in the container
Full test suite
No new failures
7,852 passed · 530 skipped · 8,417 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
15vs 22 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I'm using the latest version of Pydantic AI
- I've searched for my issue in the issue tracker before opening this issue
Description
When using OpenAIResponsesModel with providers that emit response.reasoning_text.delta events (instead of response.reasoning_summary_text.delta), the reasoning content is dropped — ThinkingPart.content remains empty (or "").
This affects providers like:
- gpt-oss via LM Studio / OpenRouter
- vLLM (both native
/v1/responsesand via litellm bridge) - Any OpenAI-compatible provider that uses
reasoning_textcontent type instead ofreasoning_summary_text
The official OpenAI Responses API only emits response.reasoning_summary_text.delta, but third-party implementations use response.reasoning_text.delta for raw chain-of-thought. The ResponseReasoningTextDeltaEvent handler was added in response to #3191, but it is incomplete — it does not pass content=chunk.delta to handle_thinking_delta().
Two code paths are affected:
- Streaming (
openai.py~line 3704):ResponseReasoningTextDeltaEventhandler callshandle_thinking_delta()withoutcontent=chunk.delta. The delta text only goes toprovider_detailsvia_make_raw_content_updater. Compare with theResponseReasoningSummaryTextDeltaEventhandler (~line 3690) which correctly passescontent=chunk.delta.
- Non-streaming (
openai.py~line 2004): Whenitem.summaryis empty butitem.contenthasreasoning_textentries,ThinkingPart(content='')is created. The raw content is only stored inprovider_details['raw_content'].
Expected: ThinkingPart.content should contain the actual reasoning text, not an empty string.
Minimal, Reproducible Example
# Simulated streaming response with reasoning_text.delta events
# (as emitted by vLLM, LM Studio with gpt-oss, or litellm bridge)
from pydantic_ai.models.openai import OpenAIResponsesModel, OpenAIResponsesStreamedResponse
# When a provider sends:
# {"type": "response.reasoning_text.delta", "item_id": "rs_123", "delta": "Let me think...", "content_index": 0}
# {"type": "response.reasoning_text.delta", "item_id": "rs_123", "delta": "The answer is 4.", "content_index": 0}
#
# The handler processes the event but NEVER passes chunk.delta as content:
#
# elif isinstance(chunk, responses.ResponseReasoningTextDeltaEvent):
# for event in self._parts_manager.handle_thinking_delta(
# vendor_part_id=chunk.item_id,
# # MISSING: content=chunk.delta,
# id=chunk.item_id,
# ...
# ):
#
# Result: ThinkingPart(content='') instead of ThinkingPart(content='Let me think... The answer is 4.')
You can verify by running the existing tests test_openai_responses_raw_cot_only, test_openai_responses_raw_cot_stream_openrouter, and test_openai_responses_raw_cot_sent_in_multiturn — all assert content='' for ThinkingPart when raw CoT is present, which is the bug.
Logfire Trace
_No Logfire trace available — issue found via curl testing against vLLM + litellm proxy._
Python, Pydantic AI & LLM client version
- Python: 3.13
- Pydantic AI: 2.5.1
- LLM provider SDK: openai>=1.99.0 (via vLLM v0.x + litellm proxy)
The two patches
pydantic_ai_slim/pydantic_ai/models/openai.pyindex 7cd80947..ccf0213b 100644@@ -2004,7 +2004,7 @@ class OpenAIResponsesModel(Model[AsyncOpenAI]): elif signature or provider_details: items.append( ThinkingPart(- content='',+ content=''.join(raw_content) if raw_content else '', id=item.id, signature=signature, provider_name=self.system,@@ -2976,7 +2976,7 @@ class OpenAIResponsesModel(Model[AsyncOpenAI]): ) openai_messages.append(reasoning_item) - if item.content:+ if item.content and item.content != ''.join(raw_content or []): # The check above guarantees that `reasoning_item` is not None assert reasoning_item is not None reasoning_item['summary'] = [@@ -3702,6 +3702,7 @@ class OpenAIResponsesStreamedResponse(StreamedResponse): # Handle raw CoT from gpt-oss models using callback pattern for event in self._parts_manager.handle_thinking_delta( vendor_part_id=chunk.item_id,+ content=chunk.delta, id=chunk.item_id, provider_name=self.provider_name, provider_details=_make_raw_content_updater(chunk.delta, chunk.content_index),tests/models/test_openai_responses.pyindex dfeb4396..5ae71c17 100644@@ -9957,7 +9957,7 @@ async def test_openai_responses_raw_cot_only(allow_model_requests: None): ModelResponse( parts=[ ThinkingPart(- content='',+ content='Let me think about this...The answer is 4.', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Let me think about this...', 'The answer is 4.']},@@ -10148,7 +10148,7 @@ async def test_openai_responses_raw_cot_stream_openrouter(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='The user asks: "What is 2+2?" They expect a straightforward answer: 4. Just answer 4.', id='rs_tmp_2kbe7x16sax', provider_name='openrouter', provider_details={@@ -10294,7 +10294,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='Raw CoT step 1Raw CoT step 2', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10331,7 +10331,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='Raw CoT step 1Raw CoT step 2', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10400,7 +10400,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='Raw CoT step 1Raw CoT step 2', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},
pydantic_ai_slim/pydantic_ai/models/openai.pyindex 7cd8094790..9f38a15fb4 100644@@ -2002,9 +2002,14 @@ class OpenAIResponsesModel(Model[AsyncOpenAI]): signature = None provider_details = {} elif signature or provider_details:+ # When there are no summaries but there is raw CoT content+ # (e.g. from gpt-oss models, vLLM, or litellm bridge), use+ # the raw content text as the ThinkingPart content so it is+ # visible to consumers, not just stored in provider_details.+ raw_content_text = ' '.join(raw_content) if raw_content else '' items.append( ThinkingPart(- content='',+ content=raw_content_text, id=item.id, signature=signature, provider_name=self.system,@@ -3699,9 +3704,12 @@ class OpenAIResponsesStreamedResponse(StreamedResponse): yield event elif isinstance(chunk, responses.ResponseReasoningTextDeltaEvent):- # Handle raw CoT from gpt-oss models using callback pattern+ # Handle raw CoT from gpt-oss models and OpenAI-compatible providers+ # (e.g. vLLM, LM Studio, litellm bridge) that use `reasoning_text` events+ # instead of `reasoning_summary_text` events. for event in self._parts_manager.handle_thinking_delta( vendor_part_id=chunk.item_id,+ content=chunk.delta, id=chunk.item_id, provider_name=self.provider_name, provider_details=_make_raw_content_updater(chunk.delta, chunk.content_index),tests/models/test_openai_responses.pyindex dfeb439621..1c9dc96fea 100644@@ -9957,7 +9957,7 @@ async def test_openai_responses_raw_cot_only(allow_model_requests: None): ModelResponse( parts=[ ThinkingPart(- content='',+ content='Let me think about this... The answer is 4.', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Let me think about this...', 'The answer is 4.']},@@ -10148,7 +10148,7 @@ async def test_openai_responses_raw_cot_stream_openrouter(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='The user asks: "What is 2+2?" They expect a straightforward answer: 4. Just answer 4.', id='rs_tmp_2kbe7x16sax', provider_name='openrouter', provider_details={@@ -10294,7 +10294,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='Raw CoT step 1 Raw CoT step 2', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10331,7 +10331,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='Raw CoT step 1 Raw CoT step 2', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},@@ -10400,7 +10400,7 @@ async def test_openai_responses_raw_cot_sent_in_multiturn(allow_model_requests: ModelResponse( parts=[ ThinkingPart(- content='',+ content='Raw CoT step 1 Raw CoT step 2', id='rs_123', provider_name='openai', provider_details={'raw_content': ['Raw CoT step 1', 'Raw CoT step 2']},
What the agent reported
I'm waiting for the fail-before-fix test run (monitored) to report. Standing by.