Backtestspydantic-airun-20260918-215936#1447

Issue #1447 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

Support for OpenAI Realtime API and equivalents (AWS Nova Sonic, etc)

Error

The harness couldn't finish this ticket.

The solver produced no patch.. No judgment was produced for this ticket. It counts as an error in the run total.

Solve

4m 3s7 turns

Wall-clock time in the container

Full test suite

Not run

The harness did not reach the test stage

Contamination audit

Unknown

See the audit notes for this run

Patch size

0vs 1,050 merged

Changed lines, ours vs merged

Description

The OpenAI Agents SDK is probably one of the first SDKs that supports voice agents OOTB and I think that it would be a nice addition to PydanticAI.

Essentially, these are native speech-to-speech models that can perform tool calling, transcription, interruption detection, etc.

The implementation differs slightly between providers (Google, AWS, OpenAI...) but the idea is similar with normal LLMs: send user input, capture all the steps that happen (tool calls, partial transcriptions) and then returning output.

OpenAI Agents SDK have implemented them using a VoicePipeline, as shown here:

https://openai.github.io/openai-agents-python/voice/quickstart/

Although probably not as pressing as supporting Prompt Caching, this is definitely a use case that people are willing to switch framework for, especially companies looking to augment customer service with AI Agent triage.

Does it sound like something that Pydantic would be interested in supporting?

References

_No response_

The two patches

Our patch+0 −0
No patch was produced.
Merged fix · PR #+1050 −0
pydantic_ai_slim/pydantic_ai/agent/__init__.pyindex 83ed8755e2..717bb2fe6e 100644@@ -77,7 +77,9 @@ if TYPE_CHECKING:      from ..builtin_tools import AbstractBuiltinTool     from ..mcp import MCPServer+    from ..models.realtime import RealtimeModel     from ..ui._web import ModelsParam+    from .voice import VoiceSession  __all__ = (     'Agent',@@ -101,6 +103,25 @@ S = TypeVar('S') NoneType = type(None)  +class _VoiceModelStub(models.Model):+    """Minimal Model stub used as RunContext.model when no text model is configured.++    Voice sessions don't use the text Model for requests, but RunContext requires one.+    Tool functions that access ctx.model will get this stub.+    """++    @property+    def model_name(self) -> str:+        return 'voice-session-stub'++    @property+    def system(self) -> str:+        return 'voice'++    async def request(self, messages: Any, model_settings: Any, model_request_parameters: Any) -> Any:+        raise NotImplementedError('Text model requests are not available in voice sessions')++ @dataclasses.dataclass(init=False) class Agent(AbstractAgent[AgentDepsT, OutputDataT]):     """Class for defining "agents" - a way to have a specific type of "conversation" with an LLM.@@ -1595,6 +1616,84 @@ class Agent(AbstractAgent[AgentDepsT, OutputDataT]):          return schema +    @asynccontextmanager+    async def voice_session(+        self,+        model: RealtimeModel,+        *,+        deps: AgentDepsT = None,+        model_settings: ModelSettings | None = None,+        instructions: str | None = None,+    ) -> AsyncIterator[VoiceSession]:+        """Open a realtime voice session using the agent's tools.++        The session connects to a speech-to-speech model and automatically+        executes tool calls using the agent's registered tools.++        Example:+        ```python+        from pydantic_ai import Agent+        from pydantic_ai.models.openai_realtime import OpenAIRealtimeModel++        agent = Agent()++        @agent.tool_plain+        def get_weather(city: str) -> str:+            return f'Sunny in {city}'++        async def main():+            rt = OpenAIRealtimeModel('gpt-4o-realtime-preview')+            async with agent.voice_session(model=rt) as session:+                await session.send_audio(b'...')+                async for event in session:+                    print(event)+        ```++        Args:+            model: The realtime model to connect to.+            deps: Dependencies to pass to tool functions.+            model_settings: Optional settings for the realtime model.+            instructions: Override the agent's instructions. If not provided, the+                agent's literal instructions are used.+        """+        from .voice import VoiceSession++        # Resolve instructions from the agent if not explicitly provided+        if instructions is None:+            literal, _funcs = self._get_instructions()+            instructions = literal or ''++        # Get tool definitions from the agent's toolset+        deps = self._get_deps(deps)+        toolset = self._get_toolset(output_toolset=None)+        async with toolset:+            # Build a minimal RunContext for tool execution+            run_context = RunContext[AgentDepsT](+                deps=deps,+                model=self._get_model(None) if self._model else _VoiceModelStub(),+                usage=_usage.RunUsage(),+            )+            tools_map = await toolset.get_tools(run_context)+            tool_defs = [t.tool_def for t in tools_map.values()]++            async def tool_runner(name: str, args: dict[str, Any]) -> str:+                tool = tools_map.get(name)+                if tool is None:+                    return f'Error: unknown tool {name!r}'+                ctx = dataclasses.replace(run_context, tool_name=name)+                try:+                    result = await toolset.call_tool(name, args, ctx, tool)+                    return str(result)+                except Exception as e:+                    return f'Error: {e}'++            async with model.connect(+                instructions=instructions,+                tools=tool_defs,+                model_settings=model_settings,+            ) as connection:+                yield VoiceSession(connection, tool_runner)+     async def __aenter__(self) -> Self:         """Enter the agent context. pydantic_ai_slim/pydantic_ai/agent/voice.pynew file mode 100644index 0000000000..3ff7299d9d@@ -0,0 +1,61 @@+"""Voice session that wraps a `RealtimeConnection` with automatic tool execution."""++from __future__ import annotations as _annotations++import json+from collections.abc import AsyncIterator, Awaitable, Callable+from typing import Any++from ..models.realtime import (+    RealtimeConnection,+    ToolCall,+    ToolCallCompleted,+    ToolCallStarted,+    VoiceSessionEvent,+)++ToolRunner = Callable[[str, dict[str, Any]], Awaitable[str]]+"""Async callable that executes a tool by name with parsed args, returning the string result."""+++class VoiceSession:+    """Wraps a `RealtimeConnection` and auto-executes tool calls.++    When iterating, `ToolCall` events from the connection are intercepted:+    a `ToolCallStarted` is emitted, the tool is executed via the provided+    ``tool_runner``, the result is sent back to the model, and a+    `ToolCallCompleted` is emitted. All other events pass through directly.+    """++    def __init__(+        self,+        connection: RealtimeConnection,+        tool_runner: ToolRunner,+    ) -> None:+        self._connection = connection+        self._tool_runner = tool_runner++    async def send_audio(self, data: bytes) -> None:+        """Forward audio data to the underlying connection."""+        await self._connection.send_audio(data)++    async def __aiter__(self) -> AsyncIterator[VoiceSessionEvent]:+        async for event in self._connection:+            if isinstance(event, ToolCall):+                yield ToolCallStarted(tool_name=event.tool_name, tool_call_id=event.tool_call_id)++                try:+                    args = json.loads(event.args) if event.args else {}+                except json.JSONDecodeError:+                    args = {}++                result = await self._tool_runner(event.tool_name, args)+                await self._connection.send_tool_result(event.tool_call_id, result)++                yield ToolCallCompleted(+                    tool_name=event.tool_name,+                    tool_call_id=event.tool_call_id,+                    result=result,+                )+            else:+                yield eventpydantic_ai_slim/pydantic_ai/models/openai_realtime.pynew file mode 100644index 0000000000..068a6b241b@@ -0,0 +1,215 @@+"""OpenAI Realtime API provider for speech-to-speech sessions.++Uses WebSocket to connect to ``wss://api.openai.com/v1/realtime`` and maps+the OpenAI-specific event protocol to the shared `RealtimeEvent` types.++Requires the ``websockets`` package::++    pip install "pydantic-ai-slim[voice]"+"""++from __future__ import annotations as _annotations++import base64+import json+import os+from collections.abc import AsyncIterator+from contextlib import asynccontextmanager+from dataclasses import dataclass++try:+    import websockets+    from websockets.asyncio.client import ClientConnection+except ImportError as _import_error:  # pragma: no cover+    raise ImportError(+        'Please install the `websockets` package to use the OpenAI Realtime model, '+        'you can use the `voice` optional group - `pip install "pydantic-ai-slim[voice]"`'+    ) from _import_error++from ..settings import ModelSettings+from ..tools import ToolDefinition+from .realtime import (+    AudioDelta,+    InputTranscript,+    RealtimeConnection,+    RealtimeEvent,+    RealtimeModel,+    SessionError,+    ToolCall,+    Transcript,+    TurnComplete,+)++DEFAULT_REALTIME_URL = 'wss://api.openai.com/v1/realtime'+++def _tool_def_to_openai(tool: ToolDefinition) -> dict[str, object]:+    """Convert a pydantic-ai ToolDefinition to OpenAI realtime tool format."""+    result: dict[str, object] = {+        'type': 'function',+        'name': tool.name,+        'parameters': tool.parameters_json_schema,+    }+    if tool.description:+        result['description'] = tool.description+    return result+++class OpenAIRealtimeConnection(RealtimeConnection):+    """Live WebSocket connection to the OpenAI Realtime API."""++    def __init__(self, ws: ClientConnection) -> None:+        self._ws = ws++    async def send_audio(self, data: bytes) -> None:+        """Send PCM16 24kHz mono audio.++        The data should be raw PCM16 little-endian bytes at 24kHz sample rate.+        """+        event = {+            'type': 'input_audio_buffer.append',+            'audio': base64.b64encode(data).decode('ascii'),+        }+        await self._ws.send(json.dumps(event))++    async def send_tool_result(self, tool_call_id: str, output: str) -> None:+        # Send the function call result+        item_event = {+            'type': 'conversation.item.create',+            'item': {+                'type': 'function_call_output',+                'call_id': tool_call_id,+                'output': output,+            },+        }+        await self._ws.send(json.dumps(item_event))+        # Trigger a new model response after providing the tool result+        await self._ws.send(json.dumps({'type': 'response.create'}))++    async def __aiter__(self) -> AsyncIterator[RealtimeEvent]:+        async for raw in self._ws:+            assert isinstance(raw, str)+            data = json.loads(raw)+            event = self._map_event(data)+            if event is not None:+                yield event++    @staticmethod+    def _map_event(data: dict[str, object]) -> RealtimeEvent | None:+        event_type = data.get('type')++        if event_type == 'response.audio.delta':+            audio_b64 = data.get('delta', '')+            assert isinstance(audio_b64, str)+            return AudioDelta(data=base64.b64decode(audio_b64))++        if event_type == 'response.audio_transcript.delta':+            delta = data.get('delta', '')+            assert isinstance(delta, str)+            return Transcript(text=delta, is_final=False)++        if event_type == 'response.audio_transcript.done':+            transcript = data.get('transcript', '')+            assert isinstance(transcript, str)+            return Transcript(text=transcript, is_final=True)++        if event_type == 'conversation.item.input_audio_transcription.completed':+            transcript = data.get('transcript', '')+            assert isinstance(transcript, str)+            return InputTranscript(text=transcript, is_final=True)++        if event_type == 'response.function_call_arguments.done':+            call_id = data.get('call_id', '')+            name = data.get('name', '')+            arguments = data.get('arguments', '{}')+            assert isinstance(call_id, str)+            assert isinstance(name, str)+            assert isinstance(arguments, str)+            return ToolCall(tool_call_id=call_id, tool_name=name, args=arguments)++        if event_type == 'response.done':+            response = data.get('response', {})+            assert isinstance(response, dict)+            interrupted = response.get('status') == 'cancelled'+            return TurnComplete(interrupted=interrupted)++        if event_type == 'error':+            error = data.get('error', {})+            assert isinstance(error, dict)+            message = error.get('message', str(error))+            assert isinstance(message, str)+            return SessionError(message=message)++        # Ignore events we don't map (session.created, session.updated, etc.)+        return None+++@dataclass+class OpenAIRealtimeModel(RealtimeModel):+    """OpenAI Realtime API model.++    Args:+        model: The model name, e.g. ``'gpt-4o-realtime-preview'``.+        api_key: OpenAI API key. Falls back to ``OPENAI_API_KEY`` env var.+        base_url: WebSocket base URL. Defaults to ``wss://api.openai.com/v1/realtime``.+        voice: Voice to use for audio output (e.g. ``'alloy'``, ``'echo'``, ``'shimmer'``).+    """++    model: str = 'gpt-4o-realtime-preview'+    api_key: str | None = None+    base_url: str = DEFAULT_REALTIME_URL+    voice: str | None = None++    @property+    def model_name(self) -> str:+        return self.model++    @asynccontextmanager+    async def connect(+        self,+        *,+        instructions: str,+        tools: list[ToolDefinition] | None = None,+        model_settings: ModelSettings | None = None,+    ) -> AsyncIterator[OpenAIRealtimeConnection]:+        api_key = self.api_key or os.environ.get('OPENAI_API_KEY', '')+        url = f'{self.base_url}?model={self.model}'++        headers = {+            'Authorization': f'Bearer {api_key}',+            'OpenAI-Beta': 'realtime=v1',+        }++        async with websockets.connect(url, additional_headers=headers) as ws:+            # Wait for session.created+            raw = await ws.recv()+            assert isinstance(raw, str)+            created = json.loads(raw)+            assert created.get('type') == 'session.created', f'Expected session.created, got {created.get("type")}'++            # Build session.update payload+            session_config: dict[str, object] = {+                'instructions': instructions,+                'input_audio_transcription': {'model': 'whisper-1'},+            }++            tool_list = tools or []+            if tool_list:+                session_config['tools'] = [_tool_def_to_openai(t) for t in tool_list]++            if self.voice:+                session_config['voice'] = self.voice++            update_event = {+                'type': 'session.update',+                'session': session_config,+            }+            await ws.send(json.dumps(update_event))++            # Wait for session.updated confirmation+            raw = await ws.recv()+            assert isinstance(raw, str)+            updated = json.loads(raw)+            assert updated.get('type') == 'session.updated', f'Expected session.updated, got {updated.get("type")}'++            yield OpenAIRealtimeConnection(ws)pydantic_ai_slim/pydantic_ai/models/realtime.pynew file mode 100644index 0000000000..1385dffabc@@ -0,0 +1,170 @@+"""Abstractions for realtime speech-to-speech models.++Providers like OpenAI Realtime, Gemini Live, and Nova Sonic offer bidirectional+streaming audio APIs. These use persistent connections (WebSocket or HTTP/2) rather+than the request-response pattern of the standard `Model` interface.++This module defines the provider-facing ABCs (`RealtimeModel`, `RealtimeConnection`)+and all event types yielded during a realtime session.+"""++from __future__ import annotations as _annotations++from abc import ABC, abstractmethod+from collections.abc import AsyncIterator+from contextlib import AbstractAsyncContextManager+from dataclasses import dataclass, field++from typing_extensions import TypeAliasType++from ..settings import ModelSettings+from ..tools import ToolDefinition++# ---------------------------------------------------------------------------+# Connection-level events (yielded by RealtimeConnection.__aiter__)+# ---------------------------------------------------------------------------+++@dataclass+class AudioDelta:+    """A chunk of audio data from the model."""++    data: bytes+++@dataclass+class Transcript:+    """Model output transcription (partial or final)."""++    text: str+    is_final: bool = False+++@dataclass+class InputTranscript:+    """Transcription of the user's speech input (partial or final)."""++    text: str+    is_final: bool = False+++@dataclass+class ToolCall:+    """The model is requesting a tool call."""++    tool_call_id: str+    tool_name: str+    args: str  # JSON string+++@dataclass+class TurnComplete:+    """The model finished its turn."""++    interrupted: bool = False+++@dataclass+class SessionError:+    """An error occurred in the realtime session."""++    message: str+++RealtimeEvent = TypeAliasType(+    'RealtimeEvent',+    AudioDelta | Transcript | InputTranscript | ToolCall | TurnComplete | SessionError,+)++# ---------------------------------------------------------------------------+# Session-level events (yielded by VoiceSession.__aiter__)+# ---------------------------------------------------------------------------+++@dataclass+class ToolCallStarted:+    """Emitted when the agent begins executing a tool call."""++    tool_name: str+    tool_call_id: str+++@dataclass+class ToolCallCompleted:+    """Emitted when the agent finishes executing a tool call."""++    tool_name: str+    tool_call_id: str+    result: str+++VoiceSessionEvent = TypeAliasType(+    'VoiceSessionEvent',+    AudioDelta | Transcript | InputTranscript | ToolCallStarted | ToolCallCompleted | TurnComplete | SessionError,+)+++# ---------------------------------------------------------------------------+# Provider ABCs+# ---------------------------------------------------------------------------+++class RealtimeConnection(ABC):+    """A live connection to a realtime model.++    Providers implement this to handle protocol-specific framing (WebSocket+    messages, HTTP/2 frames, etc.).+    """++    @abstractmethod+    async def send_audio(self, data: bytes) -> None:+        """Send an audio chunk to the model.++        The expected format (sample rate, encoding) is provider-specific.+        See each provider's documentation for details.+        """+        raise NotImplementedError++    @abstractmethod+    async def send_tool_result(self, tool_call_id: str, output: str) -> None:+        """Send the result of a tool call back to the model."""+        raise NotImplementedError++    @abstractmethod+    def __aiter__(self) -> AsyncIterator[RealtimeEvent]:+        """Iterate over events from the model."""+        raise NotImplementedError+++class RealtimeModel(ABC):+    """Abstract base for realtime speech-to-speech model providers.++    Unlike the standard `Model` ABC which uses request-response, this opens a+    persistent bidirectional connection for streaming audio in and out.+    """++    @abstractmethod+    def connect(+        self,+        *,+        instructions: str,+        tools: list[ToolDefinition] = field(default_factory=list),+        model_settings: ModelSettings | None = None,+    ) -> AbstractAsyncContextManager[RealtimeConnection]:+        """Open a connection to the realtime model.++        Args:+            instructions: System instructions for the session.+            tools: Tool definitions the model may invoke.+            model_settings: Optional provider-specific settings.++        Returns:+            An async context manager yielding a `RealtimeConnection`.+        """+        raise NotImplementedError++    @property+    @abstractmethod+    def model_name(self) -> str:+        """The model name, e.g. ``'gpt-4o-realtime'``."""+        raise NotImplementedErrorpydantic_ai_slim/pyproject.tomlindex 154836fe6b..70a1c655a8 100644@@ -107,6 +107,8 @@ cli = [     "argcomplete>=3.5.0",     "pyperclip>=1.9.0", ]+# Voice (realtime speech-to-speech)+voice = ["websockets>=14.0"] # MCP mcp = ["mcp>=1.25.0,<2.0"] # FastMCPpyproject.tomlindex 33a8254966..be53cf1f4b 100644@@ -53,6 +53,7 @@ dependencies = [ [tool.hatch.metadata.hooks.uv-dynamic-versioning.optional-dependencies] examples = ["pydantic-ai-examples=={{ version }}"] a2a = ["fasta2a>=0.4.1"]+voice = ["pydantic-ai-slim[voice]=={{ version }}"] dbos = ["pydantic-ai-slim[dbos]=={{ version }}"] prefect = ["pydantic-ai-slim[prefect]=={{ version }}"] outlines-transformers = ["pydantic-ai-slim[outlines-transformers]=={{ version }}"]tests/models/test_openai_realtime.pynew file mode 100644index 0000000000..8a19e89a56@@ -0,0 +1,198 @@+"""Tests for the OpenAI Realtime model and RealtimeConnection."""++from __future__ import annotations as _annotations++import base64+import json+from collections.abc import AsyncIterator++import pytest++from pydantic_ai.models.openai_realtime import OpenAIRealtimeConnection, OpenAIRealtimeModel+from pydantic_ai.models.realtime import (+    AudioDelta,+    InputTranscript,+    SessionError,+    ToolCall,+    Transcript,+    TurnComplete,+)++# ---------------------------------------------------------------------------+# Fake WebSocket for unit testing+# ---------------------------------------------------------------------------+++class FakeWebSocket:+    """A fake WebSocket that yields pre-configured messages."""++    def __init__(self, messages: list[str]) -> None:+        self._messages = messages+        self._sent: list[str] = []++    async def send(self, data: str) -> None:+        self._sent.append(data)++    async def recv(self) -> str:+        return self._messages.pop(0)++    def __aiter__(self) -> AsyncIterator[str]:+        return self._iter_messages()++    async def _iter_messages(self) -> AsyncIterator[str]:+        for msg in self._messages:+            yield msg+++# ---------------------------------------------------------------------------+# OpenAIRealtimeConnection._map_event tests+# ---------------------------------------------------------------------------+++class TestMapEvent:+    def test_audio_delta(self) -> None:+        audio_bytes = b'\x00\x01\x02\x03'+        data = {+            'type': 'response.audio.delta',+            'delta': base64.b64encode(audio_bytes).decode(),+        }+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, AudioDelta)+        assert event.data == audio_bytes++    def test_transcript_delta(self) -> None:+        data = {'type': 'response.audio_transcript.delta', 'delta': 'Hello'}+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, Transcript)+        assert event.text == 'Hello'+        assert event.is_final is False++    def test_transcript_done(self) -> None:+        data = {'type': 'response.audio_transcript.done', 'transcript': 'Hello world'}+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, Transcript)+        assert event.text == 'Hello world'+        assert event.is_final is True++    def test_input_transcript(self) -> None:+        data = {+            'type': 'conversation.item.input_audio_transcription.completed',+            'transcript': 'What is the weather?',+        }+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, InputTranscript)+        assert event.text == 'What is the weather?'+        assert event.is_final is True++    def test_tool_call(self) -> None:+        data = {+            'type': 'response.function_call_arguments.done',+            'call_id': 'call_123',+            'name': 'get_weather',+            'arguments': '{"city": "London"}',+        }+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, ToolCall)+        assert event.tool_call_id == 'call_123'+        assert event.tool_name == 'get_weather'+        assert event.args == '{"city": "London"}'++    def test_turn_complete(self) -> None:+        data = {'type': 'response.done', 'response': {'status': 'completed'}}+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, TurnComplete)+        assert event.interrupted is False++    def test_turn_complete_interrupted(self) -> None:+        data = {'type': 'response.done', 'response': {'status': 'cancelled'}}+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, TurnComplete)+        assert event.interrupted is True++    def test_error(self) -> None:+        data = {'type': 'error', 'error': {'message': 'rate limit exceeded'}}+        event = OpenAIRealtimeConnection._map_event(data)+        assert isinstance(event, SessionError)+        assert event.message == 'rate limit exceeded'++    def test_unknown_event_returns_none(self) -> None:+        data: dict[str, object] = {'type': 'session.created'}+        assert OpenAIRealtimeConnection._map_event(data) is None++    def test_session_updated_returns_none(self) -> None:+        data: dict[str, object] = {'type': 'session.updated'}+        assert OpenAIRealtimeConnection._map_event(data) is None+++# ---------------------------------------------------------------------------+# OpenAIRealtimeConnection.send_audio / send_tool_result+# ---------------------------------------------------------------------------+++class TestConnectionSend:+    @pytest.mark.anyio+    async def test_send_audio(self) -> None:+        ws = FakeWebSocket([])+        conn = OpenAIRealtimeConnection(ws)  # type: ignore[arg-type]++        audio = b'\xff\xfe\x00\x01'+        await conn.send_audio(audio)++        assert len(ws._sent) == 1+        msg = json.loads(ws._sent[0])+        assert msg['type'] == 'input_audio_buffer.append'+        assert base64.b64decode(msg['audio']) == audio++    @pytest.mark.anyio+    async def test_send_tool_result(self) -> None:+        ws = FakeWebSocket([])+        conn = OpenAIRealtimeConnection(ws)  # type: ignore[arg-type]++        await conn.send_tool_result('call_abc', '{"temp": 22}')++        assert len(ws._sent) == 2+        # First message: conversation.item.create+        item_msg = json.loads(ws._sent[0])+        assert item_msg['type'] == 'conversation.item.create'+        assert item_msg['item']['call_id'] == 'call_abc'+        assert item_msg['item']['output'] == '{"temp": 22}'+        # Second message: response.create+        create_msg = json.loads(ws._sent[1])+        assert create_msg['type'] == 'response.create'+++# ---------------------------------------------------------------------------+# OpenAIRealtimeConnection iteration+# ---------------------------------------------------------------------------+++class TestConnectionIteration:+    @pytest.mark.anyio+    async def test_iterates_and_maps_events(self) -> None:+        messages = [+            json.dumps({'type': 'response.audio_transcript.delta', 'delta': 'Hi'}),+            json.dumps({'type': 'response.done', 'response': {'status': 'completed'}}),+            json.dumps({'type': 'session.updated'}),  # should be skipped+        ]+        ws = FakeWebSocket(messages)+        conn = OpenAIRealtimeConnection(ws)  # type: ignore[arg-type]++        events = [event async for event in conn]+        assert len(events) == 2+        assert isinstance(events[0], Transcript)+        assert isinstance(events[1], TurnComplete)+++# ---------------------------------------------------------------------------+# OpenAIRealtimeModel.model_name+# ---------------------------------------------------------------------------+++class TestOpenAIRealtimeModel:+    def test_model_name(self) -> None:+        model = OpenAIRealtimeModel(model='gpt-4o-realtime-preview-2024-12-17')+        assert model.model_name == 'gpt-4o-realtime-preview-2024-12-17'++    def test_default_model_name(self) -> None:+        model = OpenAIRealtimeModel()+        assert model.model_name == 'gpt-4o-realtime-preview'tests/test_voice_session.pynew file mode 100644index 0000000000..4405c2be00@@ -0,0 +1,304 @@+"""Tests for VoiceSession tool dispatch."""++from __future__ import annotations as _annotations++from collections.abc import AsyncIterator+from contextlib import asynccontextmanager+from typing import Any++import pytest++from pydantic_ai import Agent+from pydantic_ai.agent.voice import VoiceSession+from pydantic_ai.models.realtime import (+    AudioDelta,+    InputTranscript,+    RealtimeConnection,+    RealtimeEvent,+    RealtimeModel,+    SessionError,+    ToolCall,+    ToolCallCompleted,+    ToolCallStarted,+    Transcript,+    TurnComplete,+)+from pydantic_ai.settings import ModelSettings+from pydantic_ai.tools import ToolDefinition++# ---------------------------------------------------------------------------+# Fake implementations for testing+# ---------------------------------------------------------------------------+++class FakeRealtimeConnection(RealtimeConnection):+    """A fake connection that yields pre-configured events."""++    def __init__(self, events: list[RealtimeEvent]) -> None:+        self._events = events+        self.sent_audio: list[bytes] = []+        self.sent_tool_results: list[tuple[str, str]] = []++    async def send_audio(self, data: bytes) -> None:+        self.sent_audio.append(data)++    async def send_tool_result(self, tool_call_id: str, output: str) -> None:+        self.sent_tool_results.append((tool_call_id, output))++    async def __aiter__(self) -> AsyncIterator[RealtimeEvent]:+        for event in self._events:+            yield event+++class FakeRealtimeModel(RealtimeModel):+    """A fake model that yields a pre-configured connection."""++    def __init__(self, connection: FakeRealtimeConnection) -> None:+        self._connection = connection+        self.last_instructions: str | None = None+        self.last_tools: list[ToolDefinition] | None = None++    @property+    def model_name(self) -> str:+        return 'fake-realtime'++    @asynccontextmanager+    async def connect(+        self,+        *,+        instructions: str,+        tools: list[ToolDefinition] | None = None,+        model_settings: ModelSettings | None = None,+    ) -> AsyncIterator[FakeRealtimeConnection]:+        self.last_instructions = instructions+        self.last_tools = tools+        yield self._connection+++# ---------------------------------------------------------------------------+# VoiceSession tests+# ---------------------------------------------------------------------------+++class TestVoiceSessionPassthrough:+    """Non-ToolCall events should pass through unchanged."""++    @pytest.mark.anyio+    async def test_audio_delta_passthrough(self) -> None:+        conn = FakeRealtimeConnection([AudioDelta(data=b'\x00\x01')])++        async def noop_runner(name: str, args: dict[str, Any]) -> str:+            raise AssertionError('should not be called')++        session = VoiceSession(conn, noop_runner)+        events = [e async for e in session]+        assert len(events) == 1+        assert isinstance(events[0], AudioDelta)+        assert events[0].data == b'\x00\x01'++    @pytest.mark.anyio+    async def test_transcript_passthrough(self) -> None:+        conn = FakeRealtimeConnection([Transcript(text='hello', is_final=True)])+        session = VoiceSession(conn, lambda n, a: None)  # type: ignore+        events = [e async for e in session]+        assert isinstance(events[0], Transcript)++    @pytest.mark.anyio+    async def test_turn_complete_passthrough(self) -> None:+        conn = FakeRealtimeConnection([TurnComplete(interrupted=True)])+        session = VoiceSession(conn, lambda n, a: None)  # type: ignore+        events = [e async for e in session]+        assert isinstance(events[0], TurnComplete)+        assert events[0].interrupted is True++    @pytest.mark.anyio+    async def test_input_transcript_passthrough(self) -> None:+        conn = FakeRealtimeConnection([InputTranscript(text='weather', is_final=False)])+        session = VoiceSession(conn, lambda n, a: None)  # type: ignore+        events = [e async for e in session]+        assert isinstance(events[0], InputTranscript)++    @pytest.mark.anyio+    async def test_session_error_passthrough(self) -> None:+        conn = FakeRealtimeConnection([SessionError(message='oops')])+        session = VoiceSession(conn, lambda n, a: None)  # type: ignore+        events = [e async for e in session]+        assert isinstance(events[0], SessionError)+        assert events[0].message == 'oops'+++class TestVoiceSessionToolDispatch:+    """ToolCall events should be intercepted and dispatched."""++    @pytest.mark.anyio+    async def test_tool_call_produces_started_and_completed(self) -> None:+        tool_call = ToolCall(tool_call_id='tc_1', tool_name='get_weather', args='{"city": "Paris"}')+        conn = FakeRealtimeConnection([tool_call])++        async def runner(name: str, args: dict[str, Any]) -> str:+            assert name == 'get_weather'+            assert args == {'city': 'Paris'}+            return 'Sunny, 22C'++        session = VoiceSession(conn, runner)+        events = [e async for e in session]++        assert len(events) == 2+        started = events[0]+        assert isinstance(started, ToolCallStarted)+        assert started.tool_name == 'get_weather'+        assert started.tool_call_id == 'tc_1'++        completed = events[1]+        assert isinstance(completed, ToolCallCompleted)+        assert completed.tool_name == 'get_weather'+        assert completed.tool_call_id == 'tc_1'+        assert completed.result == 'Sunny, 22C'++    @pytest.mark.anyio+    async def test_tool_result_sent_to_connection(self) -> None:+        tool_call = ToolCall(tool_call_id='tc_2', tool_name='search', args='{"q": "test"}')+        conn = FakeRealtimeConnection([tool_call])++        async def runner(name: str, args: dict[str, Any]) -> str:+            return 'found it'++        session = VoiceSession(conn, runner)+        _ = [e async for e in session]++        assert conn.sent_tool_results == [('tc_2', 'found it')]++    @pytest.mark.anyio+    async def test_mixed_events_and_tool_calls(self) -> None:+        events_in = [+            Transcript(text='thinking...', is_final=False),+            ToolCall(tool_call_id='tc_3', tool_name='calc', args='{"x": 1}'),+            AudioDelta(data=b'\xaa\xbb'),+            TurnComplete(),+        ]+        conn = FakeRealtimeConnection(events_in)++        async def runner(name: str, args: dict[str, Any]) -> str:+            return '42'++        session = VoiceSession(conn, runner)+        events = [e async for e in session]++        assert len(events) == 5+        assert isinstance(events[0], Transcript)+        assert isinstance(events[1], ToolCallStarted)+        assert isinstance(events[2], ToolCallCompleted)+        assert isinstance(events[3], AudioDelta)+        assert isinstance(events[4], TurnComplete)++    @pytest.mark.anyio+    async def test_invalid_json_args_default_to_empty_dict(self) -> None:+        tool_call = ToolCall(tool_call_id='tc_4', tool_name='noop', args='not json')+        conn = FakeRealtimeConnection([tool_call])++        received_args: dict[str, Any] = {}++        async def runner(name: str, args: dict[str, Any]) -> str:+            nonlocal received_args+            received_args = args+            return 'ok'++        session = VoiceSession(conn, runner)+        _ = [e async for e in session]+        assert received_args == {}+++class TestVoiceSessionSendAudio:+    @pytest.mark.anyio+    async def test_send_audio_forwards_to_connection(self) -> None:+        conn = FakeRealtimeConnection([])+        session = VoiceSession(conn, lambda n, a: None)  # type: ignore+        await session.send_audio(b'\x01\x02\x03')+        assert conn.sent_audio == [b'\x01\x02\x03']+++# ---------------------------------------------------------------------------+# Agent.voice_session integration+# ---------------------------------------------------------------------------+++class TestAgentVoiceSession:+    @pytest.mark.anyio+    async def test_agent_voice_session_wires_tools(self) -> None:+        agent: Agent[None, str] = Agent()++        @agent.tool_plain+        def greet(name: str) -> str:+            """Greet someone."""+            return f'Hello {name}!'++        tool_call = ToolCall(tool_call_id='tc_5', tool_name='greet', args='{"name": "Alice"}')+        conn = FakeRealtimeConnection([tool_call, TurnComplete()])+        fake_model = FakeRealtimeModel(conn)++        async with agent.voice_session(model=fake_model) as session:+            events = [e async for e in session]++        assert len(events) == 3+        assert isinstance(events[0], ToolCallStarted)+        assert isinstance(events[1], ToolCallCompleted)+        assert events[1].result == 'Hello Alice!'+        assert isinstance(events[2], TurnComplete)++        # Verify tools were passed to the model+        assert fake_model.last_tools is not None+        tool_names = [t.name for t in fake_model.last_tools]+        assert 'greet' in tool_names++    @pytest.mark.anyio+    async def test_agent_voice_session_passes_instructions(self) -> None:+        agent: Agent[None, str] = Agent(instructions='You are a helpful assistant.')++        conn = FakeRealtimeConnection([TurnComplete()])+        fake_model = FakeRealtimeModel(conn)++        async with agent.voice_session(model=fake_model) as session:+            _ = [e async for e in session]++        assert fake_model.last_instructions == 'You are a helpful assistant.'++    @pytest.mark.anyio+    async def test_agent_voice_session_custom_instructions(self) -> None:+        agent: Agent[None, str] = Agent(instructions='Default instructions')++        conn = FakeRealtimeConnection([TurnComplete()])+        fake_model = FakeRealtimeModel(conn)++        async with agent.voice_session(model=fake_model, instructions='Custom voice instructions') as session:+            _ = [e async for e in session]++        assert fake_model.last_instructions == 'Custom voice instructions'++    @pytest.mark.anyio+    async def test_agent_voice_session_unknown_tool(self) -> None:+        """Tool calls for unknown tools should return an error string, not crash."""+        agent: Agent[None, str] = Agent()++        tool_call = ToolCall(tool_call_id='tc_x', tool_name='nonexistent', args='{}')+        conn = FakeRealtimeConnection([tool_call, TurnComplete()])+        fake_model = FakeRealtimeModel(conn)++        async with agent.voice_session(model=fake_model) as session:+            events = [e async for e in session]++        completed = events[1]+        assert isinstance(completed, ToolCallCompleted)+        assert 'Error' in completed.result+        assert 'nonexistent' in completed.result++    @pytest.mark.anyio+    async def test_agent_voice_session_send_audio(self) -> None:+        agent: Agent[None, str] = Agent()++        conn = FakeRealtimeConnection([])+        fake_model = FakeRealtimeModel(conn)++        async with agent.voice_session(model=fake_model) as session:+            await session.send_audio(b'\xab\xcd')++        assert conn.sent_audio == [b'\xab\xcd'] 

What the agent reported

The working tree contains only the pre-existing deletions from before my session; my investigation was entirely read-only. I have enough to make a final, defensible determination without fabricating a speculative implementation.

Final Report

What the report asks for

Issue #1447 is a feature request / RFC, not a bug report. Its explicit asks are:

  1. Add native speech-to-speech ("realtime") model support à la the OpenAI Agents SDK VoicePipeline.
  2. Cover OpenAI Realtime API and equivalents (AWS Nova Sonic, Google live/bidi).
  3. Support the realtime-specific capabilities: tool calling, live transcription, interruption detection, and bidirectional user-input → intermediate-steps → output flow.
  4. It closes with a literal question — "Does it sound like something that Pydantic would be interested in supporting?" — i.e. it is soliciting a design/roadmap decision, not reporting a broken, testable behavior.

What I found (evidence)

  • No existing scaffolding. Broad case-insensitive greps for realtime, voice, speech, live, session, transcription, and bidirectional streaming across the package and docs/ return only incidental hits (test cassette names, mock files). There is nothing to fix or complete.
  • The core Model ABC is request/response-shaped, not bidirectional. pydantic_ai_slim/pydantic_ai/models/__init__.py:636 — request() (:662) returns a single ModelResponse; request_stream() (:685) yields a one-directional StreamedResponse. There is no session/interruption/live-audio abstraction, and audio is input-only (messages.py AudioUrl/BinaryContent; no audio-output content type exists). A realtime feature would require a brand-new public bidirectional abstraction — a design decision owned by the maintainers.
  • The project's own rules route this out of core. docs/contributing.md:53-62 ("Rules for adding new models"): anything "that requires more logic" should ship as a separate pydantic-ai-xxx package inheriting from the Model ABC. Multi-provider realtime S2S is squarely in that category (and network/gh access is blocked here, so upstream cannot be added anyway).

Disposition — why no code change was made

This cannot be resolved by "the smallest production-code change" backed by a regression test:

  • There is no defined, user-visible bug behavior to reproduce or assert against — the report asks for a new capability and a roadmap answer.
  • A faithful implementation is thousands of lines spanning a new bidirectional API abstraction, new audio-output content types, and per-provider client integrations (OpenAI Realtime, Nova Sonic, Google live) — the opposite of a minimal fix, and per the repo's own contribution rules it belongs in a separate package.
  • Any test I could add would either be trivial (passing before and after — explicitly forbidden by the workflow) or would only exercise a public API I invented myself, which would not match whatever the maintainers ultimately design. Per the rules, shipping such a test is "worse than no test."

I therefore did not fabricate a speculative framework, and I left the working tree unchanged (the three D entries in git status — .claude/settings.json, AGENTS.md, CLAUDE.md — pre-date my session; I created and modified nothing).

Requirements checklist

#Requirement from reportDisposition
1Native speech-to-speech / realtime model support (VoicePipeline-style)Skipped — out of scope for a minimal, test-backed fix. No abstraction exists (Model ABC is request/response only, models/__init__.py:636); requires new greenfield bidirectional API.
2Cover OpenAI Realtime + AWS Nova Sonic + Google liveSkipped. Multi-provider integration; per docs/contributing.md:53-62 such logic-heavy models ship as separate pydantic-ai-xxx packages. Network blocked, so no provider clients can be added/tested here.
3Realtime capabilities (tool calls, transcription, interruption)Skipped. Depends entirely on requirement 1's non-existent abstraction; no partial hook exists to extend.
4"Would Pydantic support this?" (roadmap question)Not implementable — a discussion question, not a code change; recorded here rather than acted on.

Recommendation

This issue should be handled as a maintainer-owned design/RFC (a new bidirectional RealtimeModel/session abstraction plus audio-output content types), with concrete provider integrations most likely delivered as separate pydantic-ai-* packages per the existing contribution rules — not as an unattended minimal patch. No test was added because none could exercise a real user-visible outcome the report describes without first building that framework, and a trivial or self-referential test would violate the workflow's rules.