Backtestspydantic-airun-20260918-215936#3976

Issue #3976 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

OpenAIResponsesModel should use developer role for SystemPromptParts by default

Worse

The merged fix was judged better than our patch.

No judgment was produced for this ticket. It counts as an error in the run total.

Solve

9m 14s3 turns

Wall-clock time in the container

Full test suite

1 new failure

3,313 passed · 163 skipped · 3,552 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

55vs 17 merged

Changed lines, ours vs merged

Question

Hi, I’m using Pydantic AI’s OpenAIResponsesModel and providing an input prompt via the instructions parameter. In the OpenAI Logs dashboard, requests generated with that code always have their instructions labeled as “System instructions” at the top.

But when I run the same instructions via the OpenAI Playground / Prompt dashboard, it shows up in the logs labeled as “Developer” at the top.

OpenAI’s latest docs also seem to emphasize “developer” instructions for app-level behavior (rather than "system"), which makes it seem like the best practice for their API is to set "role": "developer" for instructions like these, I think?

Is there a setting or hook in OpenAIResponsesModel to send these as developer instructions (or otherwise match the Playground behavior)? If not, would you consider adding one, or documenting the current behavior and recommended workaround?

Ultimately, I'd like to be able to test/optimize prompts in the playground, and then have my Pydantic AI agents in my application send requests to the API that have identical configuration as the ones I'm tweaking in the playground.

Thanks for reading!

Additional Context

pydantic-ai v1.0.18
Python 3.13.3

The two patches

Our patch+53 −2
pydantic_ai_slim/pydantic_ai/models/openai.pyindex 5365deb6..565f6e9f 100644@@ -1631,7 +1631,10 @@ class OpenAIResponsesModel(Model):             if isinstance(message, ModelRequest):                 for part in message.parts:                     if isinstance(part, SystemPromptPart):-                        openai_messages.append(responses.EasyInputMessageParam(role='system', content=part.content))+                        system_prompt_role = profile.openai_system_prompt_role or 'system'+                        openai_messages.append(+                            responses.EasyInputMessageParam(role=system_prompt_role, content=part.content)+                        )                     elif isinstance(part, UserPromptPart):                         openai_messages.append(await self._map_user_prompt(part))                     elif isinstance(part, ToolReturnPart):tests/models/test_openai_responses.pyindex 00b2276b..20dd55ce 100644@@ -48,7 +48,7 @@ from pydantic_ai.messages import ( ) from pydantic_ai.models import ModelRequestParameters from pydantic_ai.output import NativeOutput, PromptedOutput, TextOutput, ToolOutput-from pydantic_ai.profiles.openai import openai_model_profile+from pydantic_ai.profiles.openai import OpenAIModelProfile, openai_model_profile from pydantic_ai.tools import ToolDefinition from pydantic_ai.usage import RequestUsage, RunUsage @@ -8950,6 +8950,54 @@ async def test_openai_responses_system_prompts_ordering(allow_model_requests: No     )  +async def test_openai_responses_system_prompt_role_developer(allow_model_requests: None):+    """Test that the `openai_system_prompt_role` profile flag is honored by the Responses model."""+    c = response_message(+        [+            ResponseOutputMessage(+                id='msg_123',+                content=cast(list[Content], [ResponseOutputText(text='ok', type='output_text', annotations=[])]),+                role='assistant',+                status='completed',+                type='message',+            ),+        ],+    )+    mock_client = MockOpenAIResponses.create_mock(c)+    model = OpenAIResponsesModel(+        'gpt-4o',+        profile=OpenAIModelProfile(openai_system_prompt_role='developer'),+        provider=OpenAIProvider(openai_client=mock_client),+    )++    messages: list[ModelRequest | ModelResponse] = [+        ModelRequest(+            parts=[+                SystemPromptPart(content='System prompt 1'),+                SystemPromptPart(content='System prompt 2'),+                UserPromptPart(content='Hello'),+            ],+            instructions='Instructions content',+        ),+    ]++    instructions, openai_messages = await model._map_messages(  # type: ignore[reportPrivateUsage]+        messages,+        model_settings=cast(OpenAIResponsesModelSettings, {}),+        model_request_parameters=ModelRequestParameters(),+    )++    assert instructions == 'Instructions content'++    assert openai_messages == snapshot(+        [+            {'role': 'developer', 'content': 'System prompt 1'},+            {'role': 'developer', 'content': 'System prompt 2'},+            {'role': 'user', 'content': 'Hello'},+        ]+    )++ async def test_reasoning_summary_auto(allow_model_requests: None, openai_api_key: str):     model = OpenAIResponsesModel('gpt-5.2', provider=OpenAIProvider(api_key=openai_api_key))     agent = Agent(model, instructions='You are a helpful coding assistant.') 
Merged fix · PR #+13 −4
pydantic_ai_slim/pydantic_ai/models/openai.pyindex 5365deb688..bda2cb01a8 100644@@ -1401,9 +1401,10 @@ class OpenAIResponsesModel(Model):             # > Response input messages must contain the word 'json' in some form to use 'text.format' of type 'json_object'.             # Apparently they're only checking input messages for "JSON", not instructions.             assert isinstance(instructions, str)-            system_prompt_count = sum(1 for m in openai_messages if m.get('role') == 'system')+            system_prompt_role = profile.openai_system_prompt_role or 'system'+            system_prompt_count = sum(1 for m in openai_messages if m.get('role') == system_prompt_role)             openai_messages.insert(-                system_prompt_count, responses.EasyInputMessageParam(role='system', content=instructions)+                system_prompt_count, responses.EasyInputMessageParam(role=system_prompt_role, content=instructions)             )             instructions = OMIT @@ -1631,7 +1632,15 @@ class OpenAIResponsesModel(Model):             if isinstance(message, ModelRequest):                 for part in message.parts:                     if isinstance(part, SystemPromptPart):-                        openai_messages.append(responses.EasyInputMessageParam(role='system', content=part.content))+                        system_prompt_role = profile.openai_system_prompt_role+                        if system_prompt_role == 'developer':+                            openai_messages.append(+                                responses.EasyInputMessageParam(role='developer', content=part.content)+                            )+                        elif system_prompt_role == 'user':+                            openai_messages.append(responses.EasyInputMessageParam(role='user', content=part.content))+                        else:+                            openai_messages.append(responses.EasyInputMessageParam(role='system', content=part.content))                     elif isinstance(part, UserPromptPart):                         openai_messages.append(await self._map_user_prompt(part))                     elif isinstance(part, ToolReturnPart):pydantic_ai_slim/pydantic_ai/profiles/openai.pyindex 1834052a87..49f7b8e421 100644@@ -122,7 +122,7 @@ def openai_model_profile(model_name: str) -> ModelProfile:      # The o1-mini model doesn't support the `system` role, so we default to `user`.     # See https://github.com/pydantic/pydantic-ai/issues/974 for more details.-    openai_system_prompt_role = 'user' if model_name.startswith('o1-mini') else None+    openai_system_prompt_role = 'user' if model_name.startswith('o1-mini') else 'developer'      return OpenAIModelProfile(         json_schema_transformer=OpenAIJsonSchemaTransformer, 

What the agent reported

The suite is still running (no output yet). I'll wait for the completion notification.