Backtestsopenai-agents-pythonrun-20261024-184640#5191
Voice: add_audio(None) doesn't end the STT session, so the pipeline waits ~1000s for EVENT_INACTIVITY_TIMEOUT
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches send a final commit, tolerate the empty-buffer error, and wait for outstanding transcripts, but they differ in how the session actually ends. Patch B's `_finish_input` commits only when server VAD reported uncommitted speech, waits for pending `committed` items to get their transcripts (bounded by a 10 s deadline plus a 0.3 s quiet-period), and then closes the websocket itself so `_handle_events` receives `WebsocketDoneSentinel`. This is exactly the shape the issue proposes and does not depend on the server replying to anything after end of input. Patch A instead unconditionally sends `commit` + `clear` from inside `_handle_events` and only breaks once it has seen `input_audio_buffer.cleared` and all pending transcripts; it never closes the socket on its own. If the server does not answer the clear (the issue's own repro server, or any deviation in error code / event_id matching), A falls back to the full 1000 s `EVENT_INACTIVITY_TIMEOUT` and then raises an error, which is the very symptom the issue reports plus a new failure. The independent repro-server test confirms this: A still hangs past 5 s while B passes; A passes the four tests where the fake server cooperates. A also widens behaviour in risky ways beyond the ask: any `conversation.item.input_audio_transcription.failed` event now kills the whole session (previously ignored, and this applies mid-session, not just at EOF), and a server-side close after EOF becomes a hard error unless `close()` was called. B's additions are confined to end-of-input bookkeeping; its main cost is a small fixed 0.3 s grace delay on every EOF and reliance on `speech_started` to decide whether to commit, which matches the issue's suggested condition. On tests, both add real websocket-server tests through the public pipeline that would fail on base (the stream would hang) and pass after; A's matrix is broader but is tailored to A's commit/clear handshake. Correctness and robustness favour B decisively.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
11m 39s
Wall-clock time in the container
Full test suite
No new failures
80 passed · 0 skipped · 80 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
358vs 348 merged
Changed lines, ours vs merged
The issue
View on GitHubPlease read this first
- Have you read the docs? Yes.
- Have you searched for related issues? Yes. #1686 / #1687 documented
Noneas the end-of-stream signal. #4995, #4259, #4825 and #4343 cover close during setup, dispatcher start, cancellation and close errors, not end of input.
Describe the bug
StreamedAudioInput.add_audio(None) is documented as "indicates the end of the stream". With OpenAISTTModel, the pipeline doesn't actually end: result.stream() keeps waiting for about 16.7 minutes and the STT websocket stays open.
In src/agents/voice/models/openai_stt.py, _stream_audio just breaks when it reads None. It sends no input_audio_buffer.commit, doesn't close the websocket, and doesn't tell _handle_events. After that, nothing on the client side ends the session:
_handle_eventsfinishes only onWebsocketDoneSentinel(socket closed) or afterEVENT_INACTIVITY_TIMEOUT = 1000seconds without events.- So
transcribe_turns()doesn't finish,_run_multi_turnnever reachesoutput._done(), andsession_endedis never emitted.
Consequences:
async for event in result.stream()hangs for ~1000 s after the caller has said the input is over, and the websocket stays open for that time.- Because no commit is sent, a final utterance that server VAD hasn't closed yet (for example push-to-talk released mid-phrase) is never transcribed.
Debug information
- Agents SDK version:
mainat c467732e - Python version: 3.12
- OS: macOS 15
- Model:
gpt-4o-transcribeviaOpenAISTTModel; reproduced against a fake websocket, no network
Repro steps
A fake server acknowledges the session, transcribes one turn and keeps the socket open. TTS is a stub.
<details><summary>Repro script</summary>
import asyncio, json, time
from unittest.mock import patch
import numpy as np
from openai import AsyncOpenAI
from agents.voice import (
StreamedAudioInput, VoicePipeline, SingleAgentVoiceWorkflow, VoicePipelineConfig,
)
from agents.voice.models.openai_stt import OpenAISTTModel, EVENT_INACTIVITY_TIMEOUT
from agents.voice.model import TTSModel, TTSModelSettings
from agents import Agent
from agents.voice.workflow import VoiceWorkflowBase
class FakeServerWS:
def __init__(self):
self.incoming: asyncio.Queue[str] = asyncio.Queue()
self.sent: list[dict] = []
self.closed = asyncio.Event()
self.incoming.put_nowait(json.dumps({"type": "session.created"}))
async def __aenter__(self): return self
async def __aexit__(self, *a): self.closed.set()
async def send(self, payload: str):
msg = json.loads(payload)
self.sent.append(msg)
if msg["type"] == "session.update":
await self.incoming.put(json.dumps({"type": "session.updated"}))
if msg["type"] == "input_audio_buffer.append" and sum(
m["type"] == "input_audio_buffer.append" for m in self.sent) == 2:
# server VAD detects end of speech and transcribes the turn
await self.incoming.put(json.dumps({
"type": "conversation.item.input_audio_transcription.completed",
"item_id": "item_1", "transcript": "hello there"}))
async def close(self): self.closed.set()
def __aiter__(self): return self
async def __anext__(self):
get = asyncio.create_task(self.incoming.get())
closed = asyncio.create_task(self.closed.wait())
done, _ = await asyncio.wait({get, closed}, return_when=asyncio.FIRST_COMPLETED)
if closed in done:
get.cancel(); raise StopAsyncIteration
closed.cancel(); return get.result()
class EchoWorkflow(VoiceWorkflowBase):
async def run(self, transcription):
yield f"You said: {transcription}."
class SilentTTS(TTSModel):
@property
def model_name(self): return "fake-tts"
async def run(self, text, settings):
yield b"\x00\x00" * 10
async def main():
ws = FakeServerWS()
client = AsyncOpenAI(api_key="sk-test")
pipeline = VoicePipeline(
workflow=EchoWorkflow(),
stt_model=OpenAISTTModel("gpt-4o-transcribe", client),
tts_model=SilentTTS(),
config=VoicePipelineConfig(tracing_disabled=True),
)
audio = StreamedAudioInput()
with patch("websockets.connect", return_value=ws):
result = await pipeline.run(audio)
await audio.add_audio(np.zeros(2400, dtype=np.int16))
await audio.add_audio(np.zeros(2400, dtype=np.int16))
await audio.add_audio(None) # documented end-of-stream signal
t0 = time.monotonic()
events = []
async def consume():
async for ev in result.stream():
events.append(getattr(ev, "event", ev.type))
try:
await asyncio.wait_for(consume(), timeout=5)
print(f"stream finished after {time.monotonic()-t0:.2f}s: {events}")
except asyncio.TimeoutError:
print(f"EVENT_INACTIVITY_TIMEOUT = {EVENT_INACTIVITY_TIMEOUT}s")
print(f"stream still open 5s after add_audio(None); events so far: {events}")
print("sent to server after end of input:",
[m['type'] for m in ws.sent][-3:])
asyncio.run(main())
</details>
Output on main:
EVENT_INACTIVITY_TIMEOUT = 1000s
stream still open 5s after add_audio(None); events so far: ['turn_started', 'voice_stream_event_audio', 'turn_ended']
sent to server after end of input: ['session.update', 'input_audio_buffer.append', 'input_audio_buffer.append']
Expected behavior
After add_audio(None), pending speech is committed and transcribed, then the STT session closes, session_ended is emitted and result.stream() finishes.
A possible fix: on None, commit the buffer if speech was detected but not yet committed (committing an empty buffer returns a server error, which the listener currently treats as fatal), then close the websocket once the committed items have their transcripts or after a short grace period, so _handle_events receives WebsocketDoneSentinel. I'm happy to share a patch if it helps.
The two patches
src/agents/voice/models/openai_stt.pyindex 6b0c4986..8fd08535 100644@@ -29,6 +29,22 @@ from ..model import StreamedTranscriptionSession, STTModel, STTModelSettings EVENT_INACTIVITY_TIMEOUT = 1000 # Timeout for inactivity in event processing SESSION_CREATION_TIMEOUT = 10 # Timeout waiting for session.created event SESSION_UPDATE_TIMEOUT = 10 # Timeout waiting for session.updated event+# After end of input, how long to wait for the final commit and pending transcripts before closing.+END_OF_INPUT_TRANSCRIPTION_TIMEOUT = 10+# After end of input, how long the server must stay quiet before the websocket is closed, so+# events already in flight (speech detection, transcripts) are not dropped.+END_OF_INPUT_GRACE_PERIOD = 0.3++_TRANSCRIPTION_COMPLETED_EVENT_TYPES = (+ "input_audio_transcription_completed", # legacy+ "conversation.item.input_audio_transcription.completed",+)+_TRANSCRIPTION_FAILED_EVENT_TYPES = (+ "input_audio_transcription_failed", # legacy+ "conversation.item.input_audio_transcription.failed",+)+_COMMIT_EMPTY_ERROR_CODE = "input_audio_buffer_commit_empty"+_FINAL_COMMIT_EVENT_ID = "event_sdk_end_of_input_commit" DEFAULT_TURN_DETECTION = {"type": "semantic_vad"} @@ -152,6 +168,13 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): self._stored_exception: Exception | None = None self._completion_signaled = False + # End-of-input bookkeeping, updated by the listener as server events arrive.+ self._speech_uncommitted = False+ self._pending_transcript_items: set[str] = set()+ self._final_commit_sent = False+ self._awaiting_final_commit = False+ self._server_activity = asyncio.Event()+ def _get_transcription_config(self) -> dict[str, Any]: transcription_config: dict[str, Any] = {"model": self._model} if self._settings.languages is not None:@@ -213,10 +236,19 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): try: async for message in self._websocket: event = json.loads(message)+ self._server_activity.set() if event.get("type") == "error":+ if self._is_final_commit_empty_error(event):+ # The server already committed the trailing speech (or it was empty), so+ # the end-of-input commit had nothing left to commit.+ logger.debug("End-of-input commit found an empty audio buffer")+ self._awaiting_final_commit = False+ continue raise STTWebsocketConnectionError(f"Error event: {event.get('error')}") + self._track_input_buffer_event(event)+ if event.get("type") in [ "session.updated", "transcription_session.updated",@@ -233,6 +265,75 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): finally: await self._event_queue.put(WebsocketDoneSentinel()) + def _is_final_commit_empty_error(self, event: dict[str, Any]) -> bool:+ if not self._final_commit_sent:+ return False+ error = event.get("error")+ if not isinstance(error, dict) or error.get("code") != _COMMIT_EMPTY_ERROR_CODE:+ return False+ event_id = error.get("event_id")+ return event_id is None or event_id == _FINAL_COMMIT_EVENT_ID++ def _track_input_buffer_event(self, event: dict[str, Any]) -> None:+ event_type = event.get("type")+ item_id = event.get("item_id")+ if event_type == "input_audio_buffer.speech_started":+ self._speech_uncommitted = True+ elif event_type == "input_audio_buffer.committed":+ self._speech_uncommitted = False+ self._awaiting_final_commit = False+ if isinstance(item_id, str):+ self._pending_transcript_items.add(item_id)+ elif event_type in _TRANSCRIPTION_COMPLETED_EVENT_TYPES + _TRANSCRIPTION_FAILED_EVENT_TYPES:+ if isinstance(item_id, str):+ self._pending_transcript_items.discard(item_id)++ def _is_input_drained(self) -> bool:+ return not self._awaiting_final_commit and not self._pending_transcript_items++ async def _finish_input(self) -> None:+ """Handle the end-of-input signal: commit trailing speech, wait for its transcript, and+ close the websocket so the listener and event handler finish."""+ assert self._websocket is not None, "Websocket not initialized"+ deadline = monotonic() + END_OF_INPUT_TRANSCRIPTION_TIMEOUT+ while True:+ if self._speech_uncommitted and not self._final_commit_sent:+ # Speech that server VAD has not closed yet would otherwise never be transcribed.+ # Only commit when speech was detected: committing an empty buffer is an error.+ self._final_commit_sent = True+ self._awaiting_final_commit = True+ try:+ await self._websocket.send(+ json.dumps(+ {+ "type": "input_audio_buffer.commit",+ "event_id": _FINAL_COMMIT_EVENT_ID,+ }+ )+ )+ except websockets.ConnectionClosed:+ return+ except Exception as e:+ await self._output_queue.put(ErrorSentinel(e))+ raise++ self._server_activity.clear()+ drained = self._is_input_drained()+ remaining = deadline - monotonic()+ if remaining <= 0:+ logger.debug("Timed out waiting for final transcripts after end of input")+ break+ try:+ await asyncio.wait_for(+ self._server_activity.wait(),+ timeout=min(END_OF_INPUT_GRACE_PERIOD, remaining) if drained else remaining,+ )+ except asyncio.TimeoutError:+ if drained:+ break++ await self._websocket.close()+ async def _configure_session(self) -> None: assert self._websocket is not None, "Websocket not initialized" transcription_config = self._get_transcription_config()@@ -314,10 +415,7 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): raise STTWebsocketConnectionError("Error parsing events") from event.error event_type = event.get("type", "unknown")- if event_type in [- "input_audio_transcription_completed", # legacy- "conversation.item.input_audio_transcription.completed",- ]:+ if event_type in _TRANSCRIPTION_COMPLETED_EVENT_TYPES: transcript = cast(str, event.get("transcript", "")) if len(transcript) > 0: self._end_turn(transcript)@@ -345,6 +443,7 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): while True: buffer = await audio_queue.get() if buffer is None:+ await self._finish_input() break if self._trace_include_sensitive_audio_data:tests/voice/test_openai_stt.pyindex 224ff7c9..e8497016 100644@@ -4,9 +4,9 @@ import asyncio import base64 import json import logging-from collections.abc import AsyncGenerator+from collections.abc import AsyncGenerator, AsyncIterator, Callable from types import SimpleNamespace-from typing import cast+from typing import Any, cast from unittest.mock import AsyncMock, MagicMock, patch import httpx2@@ -1108,3 +1108,250 @@ async def test_stream_audio_buffers_turn_audio_only_for_audio_tracing( ) else: assert session._turn_audio_buffer == []+++async def _run_end_of_input_server(+ scenario: str,+) -> tuple[list[str], list[str], list[str]]:+ """Run a streamed STT session against a real socket that never closes on its own.++ Returns the transcripts, the client message types and the pipeline lifecycle events.+ """+ from agents.voice import VoicePipeline, VoicePipelineConfig+ from agents.voice.testing import ScriptedTTSModel++ from .pipeline_test_models import QueuedVoiceWorkflow++ received: list[str] = []+ appends = 0++ async def handle_connection(socket: ServerConnection) -> None:+ nonlocal appends+ await socket.send(json.dumps({"type": "session.created"}))+ async for message in socket:+ event = json.loads(message)+ received.append(event["type"])+ if event["type"] == "session.update":+ await socket.send(json.dumps({"type": "session.updated"}))+ elif event["type"] == "input_audio_buffer.append":+ appends += 1+ if appends == 1:+ # Server VAD closes the first utterance on its own.+ for server_event in (+ {"type": "input_audio_buffer.speech_started", "item_id": "item_1"},+ {"type": "input_audio_buffer.committed", "item_id": "item_1"},+ {+ "type": "conversation.item.input_audio_transcription.completed",+ "item_id": "item_1",+ "transcript": "hello there",+ },+ ):+ await socket.send(json.dumps(server_event))+ elif appends == 2 and scenario != "no_trailing_speech":+ # A second utterance starts but input ends before VAD closes it.+ await socket.send(+ json.dumps(+ {"type": "input_audio_buffer.speech_started", "item_id": "item_2"}+ )+ )+ elif event["type"] == "input_audio_buffer.commit":+ if scenario == "commit_empty":+ # VAD committed the trailing speech concurrently with the client's commit.+ await socket.send(+ json.dumps({"type": "input_audio_buffer.committed", "item_id": "item_2"})+ )+ await socket.send(+ json.dumps(+ {+ "type": "error",+ "error": {+ "type": "invalid_request_error",+ "code": "input_audio_buffer_commit_empty",+ "message": "buffer too small",+ "event_id": event.get("event_id"),+ },+ }+ )+ )+ else:+ await socket.send(+ json.dumps({"type": "input_audio_buffer.committed", "item_id": "item_2"})+ )+ await socket.send(+ json.dumps(+ {+ "type": "conversation.item.input_audio_transcription.completed",+ "item_id": "item_2",+ "transcript": "and goodbye",+ }+ )+ )++ async with serve(handle_connection, "127.0.0.1", 0) as server:+ port = server.sockets[0].getsockname()[1]+ async with AsyncOpenAI(+ api_key="test-key", base_url=f"http://127.0.0.1:{port}/v1"+ ) as client:+ expected_turns = 1 if scenario == "no_trailing_speech" else 2+ workflow = QueuedVoiceWorkflow([[f"reply {i}"] for i in range(expected_turns)])+ pipeline = VoicePipeline(+ workflow=workflow,+ stt_model=OpenAISTTModel("gpt-4o-transcribe", client),+ tts_model=ScriptedTTSModel([[b"\x00\x00"]] * expected_turns),+ config=VoicePipelineConfig(tracing_disabled=True),+ )+ audio = StreamedAudioInput()+ sessions: list[OpenAISTTTranscriptionSession] = []+ create_session = OpenAISTTModel.create_session++ async def capture_session(self: OpenAISTTModel, *args: Any) -> Any:+ session = await create_session(self, *args)+ sessions.append(cast(OpenAISTTTranscriptionSession, session))+ return session++ capture = patch.object(OpenAISTTModel, "create_session", capture_session)+ capture.start()+ try:+ result = await pipeline.run(audio)+ await _wait_until(lambda: bool(sessions))+ finally:+ capture.stop()+ await audio.add_audio(np.zeros(2400, dtype=np.int16))+ await audio.add_audio(np.zeros(2400, dtype=np.int16))+ if scenario != "no_trailing_speech":+ # Speech starts well before push-to-talk is released in real use.+ await _wait_until(lambda: sessions[0]._speech_uncommitted)+ await audio.add_audio(None)++ async def consume() -> list[str]:+ return [+ event.event+ async for event in result.stream()+ if event.type == "voice_stream_event_lifecycle"+ ]++ # Without honoring end of input the stream waits for EVENT_INACTIVITY_TIMEOUT.+ lifecycle = await asyncio.wait_for(consume(), 5)+ return list(workflow.transcriptions), received, lifecycle+++@pytest.mark.asyncio+async def test_end_of_input_commits_trailing_speech_and_ends_session() -> None:+ transcripts, received, lifecycle = await _run_end_of_input_server("commit")++ assert transcripts == ["hello there", "and goodbye"]+ assert received[-1] == "input_audio_buffer.commit"+ assert lifecycle[-1] == "session_ended"+++@pytest.mark.asyncio+async def test_end_of_input_tolerates_empty_commit_after_server_vad_commit() -> None:+ transcripts, received, lifecycle = await _run_end_of_input_server("commit_empty")++ assert transcripts == ["hello there", "and goodbye"]+ assert received[-1] == "input_audio_buffer.commit"+ assert lifecycle[-1] == "session_ended"+++@pytest.mark.asyncio+async def test_end_of_input_without_trailing_speech_closes_without_commit() -> None:+ transcripts, received, lifecycle = await _run_end_of_input_server("no_trailing_speech")++ assert transcripts == ["hello there"]+ assert "input_audio_buffer.commit" not in received+ assert lifecycle[-1] == "session_ended"+++@pytest.mark.asyncio+async def test_end_of_input_closes_after_timeout_when_transcript_never_arrives(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr("agents.voice.models.openai_stt.END_OF_INPUT_TRANSCRIPTION_TIMEOUT", 0.05)+ commit_received = asyncio.Event()++ async def handle_connection(socket: ServerConnection) -> None:+ await socket.send(json.dumps({"type": "session.created"}))+ async for message in socket:+ event = json.loads(message)+ if event["type"] == "session.update":+ await socket.send(json.dumps({"type": "session.updated"}))+ elif event["type"] == "input_audio_buffer.append":+ await socket.send(json.dumps({"type": "input_audio_buffer.speech_started"}))+ elif event["type"] == "input_audio_buffer.commit":+ commit_received.set()+ await socket.send(+ json.dumps({"type": "input_audio_buffer.committed", "item_id": "item_1"})+ )++ async with serve(handle_connection, "127.0.0.1", 0) as server:+ port = server.sockets[0].getsockname()[1]+ async with AsyncOpenAI(+ api_key="test-key", base_url=f"http://127.0.0.1:{port}/v1"+ ) as client:+ audio = StreamedAudioInput()+ await audio.add_audio(np.zeros(2400, dtype=np.int16))+ session = await OpenAISTTModel("gpt-4o-transcribe", client).create_session(+ audio, STTModelSettings(), False, False+ )+ assert isinstance(session, OpenAISTTTranscriptionSession)+ consumer = asyncio.create_task(_collect_turns(session.transcribe_turns()))+ await _wait_until(lambda: session._speech_uncommitted)+ await audio.add_audio(None)+ turns = await asyncio.wait_for(consumer, 5)+ assert turns == []+ assert commit_received.is_set()+++async def _collect_turns(turns: AsyncIterator[str]) -> list[str]:+ return [turn async for turn in turns]+++async def _wait_until(condition: Callable[[], bool]) -> None:+ async def poll() -> None:+ while not condition():+ await asyncio.sleep(0.001)++ await asyncio.wait_for(poll(), 2)+++@pytest.mark.asyncio+async def test_end_of_input_commits_speech_detected_during_grace_period() -> None:+ # speech_started can still be in flight when the caller ends input.+ session = OpenAISTTTranscriptionSession(+ input=StreamedAudioInput(),+ client=create_mock_openai_client(),+ model="gpt-4o-transcribe",+ settings=STTModelSettings(),+ trace_include_sensitive_data=False,+ trace_include_sensitive_audio_data=False,+ )+ mock_ws = AsyncMock()+ session._websocket = mock_ws++ async def late_server_events() -> None:+ await asyncio.sleep(0.01)+ session._track_input_buffer_event({"type": "input_audio_buffer.speech_started"})+ session._server_activity.set()+ while not mock_ws.send.await_count:+ await asyncio.sleep(0.001)+ for event in (+ {"type": "input_audio_buffer.committed", "item_id": "item_1"},+ {+ "type": "conversation.item.input_audio_transcription.completed",+ "item_id": "item_1",+ "transcript": "late words",+ },+ ):+ session._track_input_buffer_event(event)+ session._server_activity.set()++ queue: asyncio.Queue[npt.NDArray[np.int16 | np.float32] | None] = asyncio.Queue()+ await queue.put(None)+ server = asyncio.create_task(late_server_events())+ await asyncio.wait_for(session._stream_audio(queue), 2)+ await server++ sent = [json.loads(call.args[0]) for call in mock_ws.send.call_args_list]+ assert [message["type"] for message in sent] == ["input_audio_buffer.commit"]+ mock_ws.close.assert_awaited()+ await session.close()
src/agents/voice/models/openai_stt.pyindex 6b0c4986e..977594746 100644@@ -7,6 +7,7 @@ from collections.abc import AsyncIterator from dataclasses import dataclass from time import monotonic from typing import Any, cast+from uuid import uuid4 from openai import AsyncOpenAI @@ -46,6 +47,10 @@ class WebsocketDoneSentinel: pass +class _InputCompleteSentinel:+ pass++ class _ListenerError(Exception): pass @@ -136,9 +141,9 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): asyncio.Queue() ) self._websocket: websockets.ClientConnection | None = None- self._event_queue: asyncio.Queue[dict[str, Any] | ErrorSentinel | WebsocketDoneSentinel] = (- asyncio.Queue()- )+ self._event_queue: asyncio.Queue[+ dict[str, Any] | ErrorSentinel | WebsocketDoneSentinel | _InputCompleteSentinel+ ] = asyncio.Queue() self._state_queue: asyncio.Queue[dict[str, Any] | ErrorSentinel] = asyncio.Queue() self._turn_audio_buffer: list[npt.NDArray[np.int16 | np.float32]] = [] self._tracing_span: Span[TranscriptionSpanData] | None = None@@ -151,6 +156,8 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): self._connection_task: asyncio.Task[Any] | None = None self._stored_exception: Exception | None = None self._completion_signaled = False+ self._final_commit_id: str | None = None+ self._closing = False def _get_transcription_config(self) -> dict[str, Any]: transcription_config: dict[str, Any] = {"model": self._model}@@ -215,6 +222,15 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): event = json.loads(message) if event.get("type") == "error":+ error = event.get("error", {})+ if (+ self._final_commit_id is not None+ and error.get("event_id") == self._final_commit_id+ and error.get("code") == "input_audio_buffer_commit_empty"+ ):+ # VAD may already have committed the final buffer. Only this+ # specific response to our EOF commit is an expected no-op.+ continue raise STTWebsocketConnectionError(f"Error event: {event.get('error')}") if event.get("type") in [@@ -302,29 +318,74 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): raise async def _handle_events(self) -> None:+ pending_transcripts: set[str] = set()+ input_cleared = False while True: try: event = await asyncio.wait_for( self._event_queue.get(), timeout=EVENT_INACTIVITY_TIMEOUT )+ if isinstance(event, _InputCompleteSentinel):+ self._final_commit_id = uuid4().hex+ assert self._websocket is not None, "Websocket not initialized"+ await self._websocket.send(+ json.dumps(+ {"type": "input_audio_buffer.commit", "event_id": self._final_commit_id}+ )+ )+ # The clear acknowledgement follows the final commit (or its+ # empty-buffer error), even when VAD wins the commit race. It+ # ends input without cancelling already committed transcripts.+ await self._websocket.send(json.dumps({"type": "input_audio_buffer.clear"}))+ continue if isinstance(event, WebsocketDoneSentinel):+ if (+ not self._closing+ and self._final_commit_id is not None+ and (not input_cleared or pending_transcripts)+ ):+ raise STTWebsocketConnectionError(+ "Websocket closed before final transcription completed"+ ) # processed all events and websocket is done break if isinstance(event, ErrorSentinel): raise STTWebsocketConnectionError("Error parsing events") from event.error event_type = event.get("type", "unknown")+ if event_type == "input_audio_buffer.committed":+ pending_transcripts.add(event["item_id"])+ elif event_type == "input_audio_buffer.cleared":+ input_cleared = True+ elif event_type == "conversation.item.input_audio_transcription.failed":+ raise STTWebsocketConnectionError("Input audio transcription failed") if event_type in [ "input_audio_transcription_completed", # legacy "conversation.item.input_audio_transcription.completed", ]:+ if (+ event_type == "input_audio_transcription_completed"+ and "item_id" not in event+ ):+ # Legacy sessions don't identify individual completed items;+ # each completion still settles one outstanding transcript.+ if pending_transcripts:+ pending_transcripts.pop()+ else:+ pending_transcripts.discard(event.get("item_id", "")) transcript = cast(str, event.get("transcript", "")) if len(transcript) > 0: self._end_turn(transcript) self._start_turn() await self._output_queue.put(transcript)+ if self._final_commit_id is not None and input_cleared and not pending_transcripts:+ break await asyncio.sleep(0) # yield control- except asyncio.TimeoutError:+ except asyncio.TimeoutError as e:+ if self._final_commit_id is not None:+ error = STTWebsocketConnectionError("Timeout waiting for final transcription")+ await self._output_queue.put(ErrorSentinel(error))+ raise error from e # No new events for a while. Assume the session is done. break except Exception as e:@@ -345,6 +406,7 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): while True: buffer = await audio_queue.get() if buffer is None:+ await self._event_queue.put(_InputCompleteSentinel()) break if self._trace_include_sensitive_audio_data:@@ -510,6 +572,7 @@ class OpenAISTTTranscriptionSession(StreamedTranscriptionSession): raise exception_to_raise async def close(self) -> None:+ self._closing = True try: if self._websocket: await self._websocket.close()tests/voice/test_openai_stt_eof.pynew file mode 100644index 000000000..8c52a12b0@@ -0,0 +1,277 @@+"""Exercise streamed EOF through the real STT transport and public pipeline."""++from __future__ import annotations++import asyncio+import json+from collections.abc import AsyncIterator, Awaitable, Callable+from contextlib import asynccontextmanager+from typing import Any++import numpy as np+import pytest+from openai import AsyncOpenAI+from websockets.asyncio.server import ServerConnection, serve++from agents.voice import OpenAISTTModel, StreamedAudioInput, VoicePipeline, VoicePipelineConfig+from agents.voice.exceptions import STTWebsocketConnectionError+from agents.voice.models import openai_stt+from agents.voice.testing import ScriptedTTSModel, ScriptedVoiceWorkflow+++async def _send(socket: ServerConnection, event_type: str, **fields: Any) -> None:+ await socket.send(json.dumps({"type": event_type, **fields}))+++@asynccontextmanager+async def _pipeline(+ handle_input: Callable[[ServerConnection], Awaitable[None]],+ turns: int = 0,+) -> AsyncIterator[tuple[StreamedAudioInput, asyncio.Task[None], list[str], ScriptedVoiceWorkflow]]:+ socket_closed = asyncio.Event()++ async def handle_connection(socket: ServerConnection) -> None:+ try:+ await _send(socket, "session.created")+ assert json.loads(await socket.recv())["type"] == "session.update"+ await _send(socket, "session.updated")+ await handle_input(socket)+ await socket.wait_closed()+ finally:+ socket_closed.set()++ async with serve(handle_connection, "127.0.0.1", 0) as server:+ port = server.sockets[0].getsockname()[1]+ async with AsyncOpenAI(+ api_key="test-key", base_url=f"http://127.0.0.1:{port}/v1"+ ) as client:+ workflow = ScriptedVoiceWorkflow(["Reply."] * turns)+ pipeline = VoicePipeline(+ workflow=workflow,+ stt_model=OpenAISTTModel("gpt-4o-transcribe", client),+ tts_model=ScriptedTTSModel([[b"\x00\x00" * 10]] * turns),+ config=VoicePipelineConfig(tracing_disabled=True),+ )+ audio = StreamedAudioInput()+ result = await pipeline.run(audio)+ events: list[str] = []++ async def consume() -> None:+ async for event in result.stream():+ events.append(getattr(event, "event", event.type))++ consumer = asyncio.create_task(consume())+ try:+ yield audio, consumer, events, workflow+ finally:+ if not consumer.done():+ consumer.cancel()+ await asyncio.gather(consumer, return_exceptions=True)+ await asyncio.wait_for(socket_closed.wait(), 2)+ assert result.text_generation_task is not None+ assert result.text_generation_task.done()+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "mode", ["empty", "final_buffer", "vad", "vad_race", "empty_transcript", "legacy"]+)+async def test_eof_drains_transcriptions_before_finishing(mode: str) -> None:+ cleared = asyncio.Event()+ release_transcripts = asyncio.Event()+ vad_completed = asyncio.Event()+ expected = [] if mode in {"empty", "empty_transcript"} else ["Final phrase"]+ if mode == "vad_race":+ expected = ["Second phrase", "First phrase"]++ async def handle_input(socket: ServerConnection) -> None:+ if mode != "empty":+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.append"+ if mode == "vad":+ await _send(socket, "input_audio_buffer.committed", item_id="first")+ await _send(+ socket,+ "conversation.item.input_audio_transcription.completed",+ item_id="first",+ transcript="Final phrase",+ )+ vad_completed.set()+ commit = json.loads(await socket.recv())+ assert commit["type"] == "input_audio_buffer.commit"+ if mode in {"empty", "vad", "vad_race"}:+ if mode == "vad_race":+ # VAD won the commit race, but neither transcript has completed yet.+ await _send(socket, "input_audio_buffer.committed", item_id="first")+ await _send(socket, "input_audio_buffer.committed", item_id="second")+ await _send(+ socket,+ "error",+ error={+ "code": "input_audio_buffer_commit_empty",+ "event_id": commit["event_id"],+ "message": "Input buffer is empty",+ },+ )+ else:+ await _send(socket, "input_audio_buffer.committed", item_id="first")+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.clear"+ await _send(socket, "input_audio_buffer.cleared")+ cleared.set()+ if mode in {"empty", "vad"}:+ return+ await release_transcripts.wait()+ if mode == "vad_race":+ await _send(+ socket,+ "conversation.item.input_audio_transcription.completed",+ item_id="second",+ transcript="Second phrase",+ )+ if mode == "legacy":+ await _send(socket, "input_audio_transcription_completed", transcript="Final phrase")+ return+ await _send(+ socket,+ "conversation.item.input_audio_transcription.completed",+ item_id="first",+ transcript=""+ if mode == "empty_transcript"+ else "First phrase"+ if mode == "vad_race"+ else "Final phrase",+ )++ async with _pipeline(handle_input, turns=len(expected)) as (audio, consumer, events, workflow):+ try:+ if mode != "empty":+ await audio.add_audio(np.zeros(4800, dtype=np.int16))+ if mode == "vad":+ await asyncio.wait_for(vad_completed.wait(), 2)+ await audio.add_audio(None)+ await asyncio.wait_for(cleared.wait(), 2)+ if mode not in {"empty", "vad"}:+ done, _ = await asyncio.wait({consumer}, timeout=0.05)+ assert not done, "EOF must wait for the outstanding transcripts"+ release_transcripts.set()+ await asyncio.wait_for(asyncio.shield(consumer), 2)+ assert list(workflow.transcriptions) == expected+ assert events == ["turn_started", "voice_stream_event_audio", "turn_ended"] * len(+ expected+ ) + ["session_ended"]+ finally:+ release_transcripts.set()+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "outcome", ["timeout", "transcription_error", "unrelated_error", "disconnect", "cancel"]+)+async def test_eof_drain_failure_and_cancellation_close_the_session(+ outcome: str, monkeypatch+) -> None:+ draining = asyncio.Event()+ release = asyncio.Event()+ if outcome == "timeout":+ monkeypatch.setattr(openai_stt, "EVENT_INACTIVITY_TIMEOUT", 0.05)++ async def handle_input(socket: ServerConnection) -> None:+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.append"+ commit = json.loads(await socket.recv())+ assert commit["type"] == "input_audio_buffer.commit"+ await _send(socket, "input_audio_buffer.committed", item_id="first")+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.clear"+ await _send(socket, "input_audio_buffer.cleared")+ draining.set()+ await release.wait()+ if outcome == "transcription_error":+ await _send(+ socket,+ "conversation.item.input_audio_transcription.failed",+ item_id="first",+ error={"message": "Synthetic failure"},+ )+ elif outcome == "unrelated_error":+ await _send(+ socket,+ "error",+ error={"code": "input_audio_buffer_commit_empty", "event_id": "unrelated"},+ )+ elif outcome == "disconnect":+ await socket.close()++ async with _pipeline(handle_input) as (audio, consumer, events, workflow):+ try:+ await audio.add_audio(np.zeros(4800, dtype=np.int16))+ await audio.add_audio(None)+ await asyncio.wait_for(draining.wait(), 2)+ if outcome == "cancel":+ consumer.cancel()+ release.set()+ error_type = (+ asyncio.CancelledError if outcome == "cancel" else STTWebsocketConnectionError+ )+ with pytest.raises(error_type):+ await asyncio.wait_for(asyncio.shield(consumer), 2)+ assert workflow.transcriptions == ()+ finally:+ release.set()+++@pytest.mark.asyncio+async def test_explicit_close_aborts_pending_eof_without_a_provider_error() -> None:+ from agents.voice import OpenAISTTTranscriptionSession, STTModelSettings++ draining = asyncio.Event()+ socket_closed = asyncio.Event()++ async def handle_connection(socket: ServerConnection) -> None:+ try:+ await _send(socket, "session.created")+ await socket.recv()+ await _send(socket, "session.updated")+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.append"+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.commit"+ await _send(socket, "input_audio_buffer.committed", item_id="pending")+ assert json.loads(await socket.recv())["type"] == "input_audio_buffer.clear"+ await _send(socket, "input_audio_buffer.cleared")+ draining.set()+ await socket.wait_closed()+ finally:+ socket_closed.set()++ async with serve(handle_connection, "127.0.0.1", 0) as server:+ port = server.sockets[0].getsockname()[1]+ async with AsyncOpenAI(+ api_key="test-key", base_url=f"http://127.0.0.1:{port}/v1"+ ) as client:+ audio = StreamedAudioInput()+ session = await OpenAISTTModel("gpt-4o-transcribe", client).create_session(+ audio, STTModelSettings(), False, False+ )+ assert isinstance(session, OpenAISTTTranscriptionSession)++ async def consume() -> list[str]:+ return [turn async for turn in session.transcribe_turns()]++ consumer = asyncio.create_task(consume())+ try:+ await audio.add_audio(np.zeros(4800, dtype=np.int16))+ await audio.add_audio(None)+ await asyncio.wait_for(draining.wait(), 2)+ await asyncio.wait_for(session.close(), 2)+ assert await asyncio.wait_for(asyncio.shield(consumer), 2) == []+ await asyncio.wait_for(socket_closed.wait(), 2)+ assert all(+ task is not None and task.done()+ for task in (+ session._connection_task,+ session._listener_task,+ session._stream_audio_task,+ session._process_events_task,+ )+ )+ finally:+ if not consumer.done():+ consumer.cancel()+ await asyncio.gather(consumer, return_exceptions=True)+ await session.close()