Backtestsopenai-agents-pythonrun-20261024-184640#5054
Cancelling a MCP server work task does not stop the worker
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the independent test pass and preserve the CancelledServer contract, so the decision rests on root-cause fidelity and risk. Patch B follows the design the issue itself sketches: on 3.11+ it uses the official `Task.cancelling()` signal, on 3.10 it substitutes a `Task` subclass that counts cancel requests and resets per command, and it adds the missing `uncancel()` to the fallback timeout path in `_run_with_timeout_in_task` so that path no longer leaves a stale request behind. It also handles the case where the command swallows the external cancellation (post-command `cancelling()` check), settles queued commands in a `finally`, and refuses new commands once the task is done (`_enqueue`). Because it relies on `cancelling()` on 3.11+, it correctly ignores anyio-style internal cancel/uncancel cycles that the real MCP transports perform. Its tests cover external cancel, swallowed cancel, queued commands, manager bookkeeping, CancelledServer, the pre-3.11 fallback timeout path, and anyio-like self-cancellation; the one skipped test is skipped only where the behaviour is genuinely undetectable (pre-3.11), and the core tests run everywhere. Patch A instead hooks the loop's task factory to monkeypatch `task.cancel` on the instance and counts every cancel() call even on 3.11+, ignoring `cancelling()`/`uncancel()`. That counter is perturbed by any library that cancels the host task internally (anyio `fail_after`, task groups, which the MCP SDK uses), so a server-raised CancelledError that follows such internal activity would be misclassified as external and kill the worker, the exact distinction the issue asks to preserve. It does not handle a command that swallows the external cancellation, so that case reproduces the original hang. It also widens scope considerably: a new required constructor argument on `_ServerWorker`, temporary replacement of the loop's task factory, and an 'emergency cleanup' run inside the cancelled task. That cleanup is bounded only by `cleanup_timeout_seconds`; if that is None and cleanup hangs, the cancelled worker again never finishes, re-creating the issue's symptom. These additions add risk without being required by the issue. B resolves the root cause more faithfully, with less risk and well-targeted tests; A's extra machinery introduces new misclassification and hang paths.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
19m 32s
Wall-clock time in the container
Full test suite
No new failures
88 passed · 0 skipped · 88 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
332vs 417 merged
Changed lines, ours vs merged
The issue
View on GitHubDescribe the bug
Cancelling a _ServerWorker task does not stop the worker. _ServerWorker._run
(src/agents/mcp/manager.py:93-110) wraps the command it is running in except BaseException, records the
error on the command's future and continues its loop. asyncio.CancelledError is a BaseException, so an
external cancellation is consumed as if it were a command failure and the worker goes back toawait self._queue.get().
The task stays alive in the cancelling state, so is_done stays False, the manager'sadd_done_callback / _handle_worker_done bookkeeping never runs, and anything awaiting the task (including
loop shutdown) waits for a task that will not finish.
Debug information
- Agents SDK version:
main@fbf59a40 - Python version: 3.14.0
Repro steps
import asyncio
from typing import Any, cast
from agents.mcp.manager import _ServerWorker
class _BlockingServer:
def __init__(self) -> None:
self.connect_started = asyncio.Event()
async def connect(self) -> None:
self.connect_started.set()
await asyncio.sleep(3600)
async def cleanup(self) -> None:
return None
async def test_cancelling_the_worker_task_stops_the_worker() -> None:
server = _BlockingServer()
worker = _ServerWorker(cast(Any, server))
caller = asyncio.create_task(worker.connect(timeout_seconds=None))
await asyncio.wait_for(server.connect_started.wait(), timeout=1)
worker._task.cancel()
await asyncio.sleep(0.2)
caller.cancel()
try:
await caller
except BaseException:
pass
assert worker._task.done(), "the worker task survived cancellation and is still looping"
Result on main:
AssertionError: the worker task survived cancellation and is still looping
<Task cancelling name='Task-2' coro=<_ServerWorker._run() running at src/agents/mcp/manager.py:95>
Expected behavior
Cancelling the worker task ends it.
Why this is not a one-line re-raise
A server's own connect/cleanup may raise CancelledError deliberately — CancelledServer intests/mcp/test_mcp_server_manager.py, together with suppress_cancelled_error, depends on that error
reaching the caller through the command future. So the two cases need separating: an error raised *by the
command* (settle the future, keep looping) versus a cancellation of the worker itself (settle the future,
then stop).
asyncio.current_task().cancelling() > 0 distinguishes them on 3.11+, but the fallback timeout path in_run_with_timeout_in_task (manager.py:135-148) calls task.cancel() and converts the result toTimeoutError without a matching uncancel(), which would leave a pending cancellation request behind on
that path. cancelling() also does not exist on 3.10, which the project still supports.
Happy to send a PR with tests once you decide which behaviour you want.
The two patches
src/agents/mcp/manager.pyindex 733e7443..7ea5310f 100644@@ -2,7 +2,8 @@ from __future__ import annotations import asyncio import math-from collections.abc import Awaitable, Callable, Iterable+import sys+from collections.abc import Awaitable, Callable, Coroutine, Iterable from contextlib import AbstractAsyncContextManager from dataclasses import dataclass from typing import Any@@ -27,6 +28,66 @@ def _validate_lifecycle_timeout(timeout_seconds: float | None, *, field_name: st return timeout_seconds +if sys.version_info >= (3, 11):++ def _create_worker_task(coro: Coroutine[Any, Any, None]) -> asyncio.Task[None]:+ return asyncio.create_task(coro)++ def _reset_cancellation_requests(task: asyncio.Task[Any]) -> None:+ return None++ def _cancellation_requested(task: asyncio.Task[Any], exc: BaseException | None) -> bool:+ """Return whether a cancellation of the task was requested and is still pending."""+ return task.cancelling() > 0++else:++ class _CancelTrackingTask(asyncio.Task): # type: ignore[type-arg]+ """Track pending cancellation requests, like Task.cancelling() on Python 3.11+."""++ def __init__(self, coro: Coroutine[Any, Any, None]) -> None:+ super().__init__(coro)+ self._cancel_requests = 0++ def cancel(self, msg: Any = None) -> bool:+ if self.done():+ return False+ self._cancel_requests += 1+ return super().cancel(msg)++ def cancelling(self) -> int:+ return self._cancel_requests++ def uncancel(self) -> int:+ if self._cancel_requests > 0:+ self._cancel_requests -= 1+ return self._cancel_requests++ def _create_worker_task(coro: Coroutine[Any, Any, None]) -> asyncio.Task[None]:+ return _CancelTrackingTask(coro)++ def _reset_cancellation_requests(task: asyncio.Task[Any]) -> None:+ if isinstance(task, _CancelTrackingTask):+ task._cancel_requests = 0++ def _cancellation_requested(task: asyncio.Task[Any], exc: BaseException | None) -> bool:+ """Return whether a cancellation of the task was requested and is still pending."""+ # Libraries such as anyio cancel the host task for their own cancel scopes and+ # cannot withdraw that request before Python 3.11, so a request count alone does+ # not prove an external cancellation. Only trust it when CancelledError surfaced.+ return (+ isinstance(exc, asyncio.CancelledError)+ and isinstance(task, _CancelTrackingTask)+ and task.cancelling() > 0+ )+++def _withdraw_cancellation(task: asyncio.Task[Any]) -> None:+ uncancel = getattr(task, "uncancel", None)+ if uncancel is not None:+ uncancel()++ @dataclass class _ServerCommand: action: str@@ -38,7 +99,7 @@ class _ServerWorker: def __init__(self, server: MCPServer) -> None: self._server = server self._queue: asyncio.Queue[_ServerCommand] = asyncio.Queue()- self._task = asyncio.create_task(self._run())+ self._task = _create_worker_task(self._run()) self._cleanup_future: asyncio.Future[None] | None = None @property@@ -69,7 +130,7 @@ class _ServerWorker: if self._cleanup_future is None: loop = asyncio.get_running_loop() self._cleanup_future = loop.create_future()- self._queue.put_nowait(+ self._enqueue( _ServerCommand( action="cleanup", timeout_seconds=timeout_seconds,@@ -85,29 +146,58 @@ class _ServerWorker: async def _submit(self, action: str, timeout_seconds: float | None) -> None: loop = asyncio.get_running_loop() future: asyncio.Future[None] = loop.create_future()- self._queue.put_nowait(- _ServerCommand(action=action, timeout_seconds=timeout_seconds, future=future)- )+ self._enqueue(_ServerCommand(action=action, timeout_seconds=timeout_seconds, future=future)) await future + def _enqueue(self, command: _ServerCommand) -> None:+ if self._task.done():+ # Nothing will ever process the command once the worker has stopped.+ command.future.cancel()+ return+ self._queue.put_nowait(command)++ def _cancel_pending_commands(self) -> None:+ while not self._queue.empty():+ self._queue.get_nowait().future.cancel()+ async def _run(self) -> None:- while True:- command = await self._queue.get()- should_exit = command.action == "cleanup"- try:- if command.action == "connect":- await _run_with_timeout_in_task(self._server.connect, command.timeout_seconds)- elif command.action == "cleanup":- await _run_with_timeout_in_task(self._server.cleanup, command.timeout_seconds)- else:- raise ValueError(f"Unknown command: {command.action}")- if not command.future.cancelled():- command.future.set_result(None)- except BaseException as exc:- if not command.future.cancelled():- command.future.set_exception(exc)- if should_exit:- return+ try:+ while True:+ command = await self._queue.get()+ should_exit = command.action == "cleanup"+ _reset_cancellation_requests(self._task)+ try:+ if command.action == "connect":+ await _run_with_timeout_in_task(+ self._server.connect, command.timeout_seconds+ )+ elif command.action == "cleanup":+ await _run_with_timeout_in_task(+ self._server.cleanup, command.timeout_seconds+ )+ else:+ raise ValueError(f"Unknown command: {command.action}")+ if not command.future.cancelled():+ command.future.set_result(None)+ except BaseException as exc:+ if _cancellation_requested(self._task, exc):+ # The worker task itself was cancelled: settle the command and stop.+ command.future.cancel()+ if isinstance(exc, asyncio.CancelledError):+ raise+ raise asyncio.CancelledError() from exc+ # The command raised (possibly CancelledError on its own): report it to+ # the caller and keep serving commands.+ if not command.future.cancelled():+ command.future.set_exception(exc)+ if should_exit:+ return+ if _cancellation_requested(self._task, None):+ # The command swallowed the worker's cancellation; honour it now+ # (only detectable on Python 3.11+).+ raise asyncio.CancelledError()+ finally:+ self._cancel_pending_commands() async def _run_with_timeout_in_task(@@ -146,6 +236,10 @@ async def _run_with_timeout_in_task( raise finally: handle.cancel()+ if timed_out:+ # Withdraw our own cancellation request so it is not mistaken for an+ # external cancellation of the task.+ _withdraw_cancellation(task) class MCPServerManager(AbstractAsyncContextManager["MCPServerManager"]):tests/mcp/test_mcp_server_manager.pyindex 40476867..c619a5e4 100644@@ -1,5 +1,6 @@ import asyncio import logging+import sys from collections.abc import Awaitable, Callable from typing import Any, cast @@ -1280,3 +1281,194 @@ async def test_manager_restores_one_shot_iterable_servers_after_a_failed_connect # drop_failed_servers=False keeps failed servers active, so the restored list must match # what an equivalent list argument produces. assert manager.active_servers == [server]+++class BlockingConnectServer(TaskBoundServer):+ def __init__(self, *, swallow_cancellation: bool = False) -> None:+ super().__init__()+ self.connect_started = asyncio.Event()+ self.swallow_cancellation = swallow_cancellation++ async def connect(self) -> None:+ self._connect_task = asyncio.current_task()+ self.connect_started.set()+ try:+ await asyncio.sleep(3600)+ except asyncio.CancelledError:+ if not self.swallow_cancellation:+ raise+++async def _wait_until_done(task: asyncio.Task[Any]) -> None:+ await asyncio.wait({task}, timeout=1)+++@pytest.mark.asyncio+async def test_cancelling_the_worker_task_stops_the_worker() -> None:+ server = BlockingConnectServer()+ worker = manager_module._ServerWorker(server)++ caller = asyncio.create_task(worker.connect(timeout_seconds=None))+ await asyncio.wait_for(server.connect_started.wait(), timeout=1)++ worker._task.cancel()+ await _wait_until_done(worker._task)++ assert worker._task.done(), "the worker task survived cancellation and is still looping"+ assert worker._task.cancelled()+ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(caller, timeout=1)+++@pytest.mark.asyncio+@pytest.mark.skipif(+ sys.version_info < (3, 11),+ reason="a swallowed cancellation is indistinguishable from a library's own before 3.11",+)+async def test_cancelling_the_worker_task_stops_the_worker_when_connect_swallows_it() -> None:+ server = BlockingConnectServer(swallow_cancellation=True)+ worker = manager_module._ServerWorker(server)++ caller = asyncio.create_task(worker.connect(timeout_seconds=None))+ await asyncio.wait_for(server.connect_started.wait(), timeout=1)++ worker._task.cancel()+ await _wait_until_done(worker._task)++ assert worker._task.cancelled()+ # The command itself completed before the worker honoured the cancellation.+ await asyncio.wait_for(caller, timeout=1)+++@pytest.mark.asyncio+async def test_cancelling_the_worker_task_settles_queued_and_later_commands() -> None:+ server = BlockingConnectServer()+ worker = manager_module._ServerWorker(server)++ first = asyncio.create_task(worker.connect(timeout_seconds=None))+ await asyncio.wait_for(server.connect_started.wait(), timeout=1)+ queued_cleanup = asyncio.create_task(worker.cleanup(timeout_seconds=None))+ await asyncio.sleep(0)++ worker._task.cancel()+ await _wait_until_done(worker._task)++ assert worker._task.done()+ for caller in (first, queued_cleanup):+ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(caller, timeout=1)+ assert worker.cleanup_error is None+ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(worker.connect(timeout_seconds=None), timeout=1)+++@pytest.mark.asyncio+async def test_manager_discards_parallel_worker_whose_task_was_cancelled() -> None:+ server = BlockingConnectServer()+ manager = MCPServerManager([server], connect_in_parallel=True, connect_timeout_seconds=None)++ connect = asyncio.create_task(manager.connect_all())+ await asyncio.wait_for(server.connect_started.wait(), timeout=1)+ worker = manager._workers[server]++ worker._task.cancel()+ await asyncio.wait_for(connect, timeout=1)++ assert worker.is_done+ assert server not in manager._workers+ assert manager.failed_servers == [server]+ assert isinstance(manager.errors[server], asyncio.CancelledError)+ await manager.cleanup_all()+++@pytest.mark.asyncio+async def test_worker_keeps_serving_after_server_raises_cancelled_error() -> None:+ server = CancelledServer()+ worker = manager_module._ServerWorker(server)++ with pytest.raises(asyncio.CancelledError):+ await worker.connect(timeout_seconds=None)++ assert not worker.is_done+ await asyncio.wait_for(worker.cleanup(timeout_seconds=None), timeout=1)+ await _wait_until_done(worker._task)+ assert worker.is_done+ assert not worker._task.cancelled()+ assert server.cleanup_calls == 1+++@pytest.mark.asyncio+async def test_worker_keeps_serving_after_fallback_timeout(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ # Exercise the pre-3.11 timeout path, which cancels the worker task to time out.+ monkeypatch.delattr(asyncio, "timeout", raising=False)+ server = BlockingConnectServer()+ worker = manager_module._ServerWorker(server)++ with pytest.raises(asyncio.TimeoutError):+ await asyncio.wait_for(worker.connect(timeout_seconds=0.05), timeout=1)++ assert not worker.is_done+ await asyncio.wait_for(worker.cleanup(timeout_seconds=None), timeout=1)+ await _wait_until_done(worker._task)+ assert worker.is_done+ assert not worker._task.cancelled()+ assert server.cleaned is True+++class SelfCancellingConnectServer(TaskBoundServer):+ """Cancel its own task internally and catch it, the way anyio cancel scopes do."""++ def __init__(self, *, error: BaseException | None = None) -> None:+ super().__init__()+ self.error = error+ self.connect_calls = 0++ async def connect(self) -> None:+ self.connect_calls += 1+ self._connect_task = asyncio.current_task()+ task = asyncio.current_task()+ assert task is not None+ task.cancel()+ try:+ await asyncio.sleep(3600)+ except asyncio.CancelledError:+ # Like anyio, only withdraw the request where Task.uncancel() exists.+ if sys.version_info >= (3, 11):+ task.uncancel()+ if self.error is not None:+ raise self.error+++@pytest.mark.asyncio+async def test_worker_reports_error_after_server_handles_its_own_cancellation() -> None:+ server = SelfCancellingConnectServer(error=RuntimeError("connect failed"))+ worker = manager_module._ServerWorker(server)++ with pytest.raises(RuntimeError, match="connect failed"):+ await asyncio.wait_for(worker.connect(timeout_seconds=None), timeout=1)++ assert not worker.is_done+ server.error = None+ await asyncio.wait_for(worker.connect(timeout_seconds=None), timeout=1)+ assert not worker.is_done+ await asyncio.wait_for(worker.cleanup(timeout_seconds=None), timeout=1)+ await _wait_until_done(worker._task)+ assert not worker._task.cancelled()+ assert server.connect_calls == 2+ assert server.cleaned is True+++@pytest.mark.asyncio+async def test_worker_keeps_serving_after_server_handles_its_own_cancellation() -> None:+ server = SelfCancellingConnectServer()+ worker = manager_module._ServerWorker(server)++ await asyncio.wait_for(worker.connect(timeout_seconds=None), timeout=1)++ assert not worker.is_done+ await asyncio.wait_for(worker.cleanup(timeout_seconds=None), timeout=1)+ await _wait_until_done(worker._task)+ assert not worker._task.cancelled()+ assert server.cleaned is True
src/agents/mcp/manager.pyindex 733e74436..cb2151117 100644@@ -5,7 +5,7 @@ import math from collections.abc import Awaitable, Callable, Iterable from contextlib import AbstractAsyncContextManager from dataclasses import dataclass-from typing import Any+from typing import Any, cast from ..logger import log_tool_action_debug, log_tool_action_error, logger from ._logging import get_mcp_server_log_message@@ -35,12 +35,77 @@ class _ServerCommand: class _ServerWorker:- def __init__(self, server: MCPServer) -> None:+ def __init__(+ self,+ server: MCPServer,+ cleanup_timeout_seconds_fn: Callable[[], float | None],+ ) -> None: self._server = server+ # The emergency cleanup below is a cleanup, not a connect, so it must be+ # bounded by the manager cleanup timeout rather than by the interrupted+ # command's own timeout. The callable is read at cancel time so runtime+ # reassignment of the manager property stays honored.+ self._cleanup_timeout_seconds_fn = cleanup_timeout_seconds_fn self._queue: asyncio.Queue[_ServerCommand] = asyncio.Queue()- self._task = asyncio.create_task(self._run())+ # The worker needs to tell an external cancel from a CancelledError a+ # server raised on its own. Task.cancelling() covers that only on+ # Python 3.11+, and this package still supports 3.10, so every cancel()+ # call on the worker task is counted here.+ self._cancel_requests = 0+ self._task: asyncio.Task[None] = self._create_task() self._cleanup_future: asyncio.Future[None] | None = None + def _create_task(self) -> asyncio.Task[None]:+ # Route worker creation through the event loop's task factory so+ # applications supervising or instrumenting tasks through+ # set_task_factory still see this worker. The temporary factory keeps+ # the app factory in charge of the task type and only adds the cancel+ # counting on top, because loop shutdown and supervisors cancel the+ # task object directly and would otherwise bypass the count.+ loop = asyncio.get_running_loop()+ app_factory = loop.get_task_factory()++ def factory(+ loop: asyncio.AbstractEventLoop, coro: Any, **kwargs: Any+ ) -> asyncio.Task[None]:+ task: asyncio.Task[None] = (+ cast(asyncio.Task[None], app_factory(loop, coro, **kwargs))+ if app_factory+ else cast(asyncio.Task[None], asyncio.Task(coro, loop=loop, **kwargs))+ )+ original_cancel = task.cancel++ def counting_cancel(msg: Any = None) -> bool:+ self._cancel_requests += 1+ return bool(original_cancel(msg))++ task.cancel = counting_cancel # type: ignore[method-assign]+ return task++ # The typeshed _TaskFactory protocol is a private generic stub; this+ # factory matches its runtime call shape exactly.+ loop.set_task_factory(factory) # type: ignore[arg-type]+ try:+ return asyncio.create_task(self._run())+ finally:+ loop.set_task_factory(app_factory)++ def _record_cleanup_failure(self, exc: BaseException) -> None:+ """Record an emergency cleanup failure on the cleanup result future.++ The done callback reads the cleanup_error property through that+ future, so a recorded failure keeps the worker registered instead of+ letting the manager treat the server as cleanly stopped. A later+ cleanup() call awaits the same settled future and surfaces the failure+ to the manager, which keeps reconnect from starting a second+ connection over a transport that was never successfully cleaned up.+ """+ if self._cleanup_future is None:+ self._cleanup_future = asyncio.get_running_loop().create_future()+ if self._cleanup_future.done():+ return+ self._cleanup_future.set_exception(exc)+ @property def is_done(self) -> bool: return self._task.done()@@ -91,23 +156,67 @@ class _ServerWorker: await future async def _run(self) -> None:- while True:- command = await self._queue.get()- should_exit = command.action == "cleanup"- try:- if command.action == "connect":- await _run_with_timeout_in_task(self._server.connect, command.timeout_seconds)- elif command.action == "cleanup":- await _run_with_timeout_in_task(self._server.cleanup, command.timeout_seconds)- else:- raise ValueError(f"Unknown command: {command.action}")- if not command.future.cancelled():- command.future.set_result(None)- except BaseException as exc:- if not command.future.cancelled():- command.future.set_exception(exc)- if should_exit:- return+ try:+ while True:+ command = await self._queue.get()+ should_exit = command.action == "cleanup"+ cancellation_baseline = self._cancel_requests+ try:+ if command.action == "connect":+ await _run_with_timeout_in_task(+ self._server.connect, command.timeout_seconds+ )+ elif command.action == "cleanup":+ await _run_with_timeout_in_task(+ self._server.cleanup, command.timeout_seconds+ )+ else:+ raise ValueError(f"Unknown command: {command.action}")+ if not command.future.cancelled():+ command.future.set_result(None)+ except BaseException as exc:+ external_cancel = (+ isinstance(exc, asyncio.CancelledError)+ and self._cancel_requests > cancellation_baseline+ )+ if external_cancel and command.action != "cleanup":+ # The worker task itself was cancelled while the command+ # was running. A partially acquired connection from the+ # interrupted command still needs cleanup, and this task+ # owns it (some transports require cleanup in the same+ # task), so clean up here before the caller is released.+ # The manager cleanup timeout bounds the emergency+ # cleanup, not the interrupted command's own timeout.+ try:+ await _run_with_timeout_in_task(+ self._server.cleanup, self._cleanup_timeout_seconds_fn()+ )+ except BaseException as cleanup_exc:+ # A failed or timed-out emergency cleanup must not be+ # swallowed silently. Recording it on the cleanup+ # result future keeps the worker registered, so a+ # later reconnect surfaces the failure and never+ # starts a second connection over the transport+ # that was never cleaned up.+ self._record_cleanup_failure(cleanup_exc)+ if not command.future.cancelled():+ command.future.set_exception(exc)+ if external_cancel:+ # The future carries the error to the caller; re-raise+ # here to end the task and honor the cancellation.+ raise+ if should_exit:+ return+ finally:+ # The worker loop has ended, so commands still queued will never run.+ # Settle their futures so waiting callers do not hang forever.+ while True:+ try:+ pending = self._queue.get_nowait()+ except asyncio.QueueEmpty:+ break+ if not pending.future.done():+ pending.future.cancel() async def _run_with_timeout_in_task(@@ -535,7 +644,10 @@ class MCPServerManager(AbstractAsyncContextManager["MCPServerManager"]): self._discard_worker(server, worker) worker = self._workers.get(server) if worker is None:- worker = _ServerWorker(server=server)+ worker = _ServerWorker(+ server=server,+ cleanup_timeout_seconds_fn=lambda: self.cleanup_timeout_seconds,+ ) self._workers[server] = worker worker.add_done_callback(lambda _task: self._handle_worker_done(server, worker)) return workertests/mcp/test_mcp_server_manager.pyindex 40476867a..c5376c58e 100644@@ -1,6 +1,6 @@ import asyncio import logging-from collections.abc import Awaitable, Callable+from collections.abc import Awaitable, Callable, Coroutine from typing import Any, cast import pytest@@ -1280,3 +1280,264 @@ async def test_manager_restores_one_shot_iterable_servers_after_a_failed_connect # drop_failed_servers=False keeps failed servers active, so the restored list must match # what an equivalent list argument produces. assert manager.active_servers == [server]+++@pytest.mark.asyncio+async def test_worker_task_cancellation_stops_worker_and_fails_caller() -> None:+ # This test pins an exact cancellation boundary: a connect that hangs until the+ # worker task itself is cancelled. The scripted utilities cannot model a worker+ # task receiving an external cancel, so a local double is used here.+ class HangingConnectServer:+ def __init__(self) -> None:+ self.connect_started = asyncio.Event()++ async def connect(self) -> None:+ self.connect_started.set()+ await asyncio.sleep(3600)++ async def cleanup(self) -> None:+ return None++ server = HangingConnectServer()+ worker = manager_module._ServerWorker(cast(Any, server), lambda: None)++ caller = asyncio.create_task(worker.connect(timeout_seconds=None))+ await asyncio.wait_for(server.connect_started.wait(), timeout=TEST_TIMEOUT_SECONDS)++ worker._task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(caller, timeout=TEST_TIMEOUT_SECONDS)++ assert worker._task.done()+ assert worker._task.cancelled()+++@pytest.mark.asyncio+async def test_worker_survives_server_raised_cancelled_error_and_serves_next_command() -> None:+ # This test pins the CancelledServer contract: a server raising CancelledError on+ # its own must reach the caller through the command future without ending the+ # worker, because later lifecycle commands still rely on that worker.+ class SelfCancellingThenOkServer:+ def __init__(self) -> None:+ self.connect_calls = 0++ async def connect(self) -> None:+ self.connect_calls += 1+ if self.connect_calls == 1:+ raise asyncio.CancelledError()++ async def cleanup(self) -> None:+ return None++ server = SelfCancellingThenOkServer()+ worker = manager_module._ServerWorker(cast(Any, server), lambda: None)++ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(worker.connect(timeout_seconds=None), timeout=TEST_TIMEOUT_SECONDS)++ assert not worker.is_done++ await asyncio.wait_for(worker.connect(timeout_seconds=None), timeout=TEST_TIMEOUT_SECONDS)+ assert server.connect_calls == 2++ await asyncio.wait_for(worker.cleanup(timeout_seconds=None), timeout=TEST_TIMEOUT_SECONDS)+ assert worker.is_done+++@pytest.mark.asyncio+async def test_worker_cancel_settles_commands_queued_behind_the_in_flight_one() -> None:+ # An external cancel ends the worker, so commands still queued behind the+ # in-flight one will never run. Their futures must be settled on the way out,+ # or the callers awaiting those futures hang forever.+ class BlockingConnectServer:+ def __init__(self) -> None:+ self.connect_started = asyncio.Event()+ self.cleanup_calls = 0++ async def connect(self) -> None:+ self.connect_started.set()+ await asyncio.sleep(3600)++ async def cleanup(self) -> None:+ self.cleanup_calls += 1++ server = BlockingConnectServer()+ worker = manager_module._ServerWorker(cast(Any, server), lambda: None)++ connect_caller = asyncio.create_task(worker.connect(timeout_seconds=None))+ await asyncio.wait_for(server.connect_started.wait(), timeout=TEST_TIMEOUT_SECONDS)+ cleanup_caller = asyncio.create_task(worker.cleanup(timeout_seconds=None))+ await asyncio.sleep(0)++ worker._task.cancel()++ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(cleanup_caller, timeout=TEST_TIMEOUT_SECONDS)+ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(connect_caller, timeout=TEST_TIMEOUT_SECONDS)++ assert worker._task.done()+ assert worker._task.cancelled()+++@pytest.mark.asyncio+async def test_worker_cancel_cleans_up_partially_acquired_connection() -> None:+ # An external cancel during a blocking connect must not leak the partially+ # acquired connection: the worker owns it and is the only task allowed to+ # clean it up, so the cleanup has to run in that task before the worker+ # terminates.+ class BlockingPartialConnectServer(TaskBoundServer):+ def __init__(self) -> None:+ super().__init__()+ self.connect_started = asyncio.Event()+ self.cleanup_calls = 0++ async def connect(self) -> None:+ self._connect_task = asyncio.current_task()+ self.connect_started.set()+ await asyncio.sleep(3600)++ async def cleanup(self) -> None:+ self.cleanup_calls += 1+ await super().cleanup()++ server = BlockingPartialConnectServer()+ manager = MCPServerManager([server], connect_in_parallel=True)++ connect_all = asyncio.create_task(manager.connect_all())+ await asyncio.wait_for(server.connect_started.wait(), timeout=TEST_TIMEOUT_SECONDS)+ manager._workers[server]._task.cancel()+ await asyncio.wait_for(connect_all, timeout=TEST_TIMEOUT_SECONDS)++ assert server.cleaned is True+ assert server.cleanup_calls == 1+ await asyncio.wait_for(manager.cleanup_all(), timeout=TEST_TIMEOUT_SECONDS)+ assert server.cleanup_calls == 1+++@pytest.mark.asyncio+async def test_worker_emergency_cleanup_uses_the_cleanup_timeout() -> None:+ # An external cancel during a connect must bound the emergency cleanup by+ # the manager cleanup timeout, not by the interrupted connect command's own+ # timeout: when connect_timeout_seconds is None the command carries no+ # bound, and an unbounded emergency cleanup can hang the worker (and loop+ # shutdown) forever, while a shorter connect timeout could cut a legitimate+ # cleanup short.+ class HangingCleanupServer:+ def __init__(self) -> None:+ self.connect_started = asyncio.Event()+ self.cleanup_started = asyncio.Event()+ self.cleanup_cancelled = False++ async def connect(self) -> None:+ self.connect_started.set()+ await asyncio.sleep(3600)++ async def cleanup(self) -> None:+ self.cleanup_started.set()+ try:+ await asyncio.sleep(3600)+ except asyncio.CancelledError:+ self.cleanup_cancelled = True+ raise++ server = HangingCleanupServer()+ worker = manager_module._ServerWorker(cast(Any, server), lambda: 0.2)++ caller = asyncio.create_task(worker.connect(timeout_seconds=None))+ await asyncio.wait_for(server.connect_started.wait(), timeout=TEST_TIMEOUT_SECONDS)+ worker._task.cancel()++ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(caller, timeout=TEST_TIMEOUT_SECONDS)+ # the caller settles only after the emergency cleanup is bounded and done,+ # so the worker must already be terminated here; awaiting a cancelled task+ # would re-raise CancelledError in the test task instead+ assert worker._task.done()+ assert worker._task.cancelled()+ assert server.cleanup_started.is_set()+ assert server.cleanup_cancelled is True+++@pytest.mark.asyncio+@pytest.mark.parametrize("mode", ["raise", "timeout"])+async def test_manager_reconnect_does_not_retry_after_failed_emergency_cleanup(+ mode: str,+) -> None:+ # A manager-level regression for the emergency cleanup result: an external+ # cancel during a partially started connect runs the emergency cleanup in+ # the worker task, and when that cleanup raises or times out the failure+ # must reach the existing cleanup result mechanism. The done callback then+ # keeps the worker instead of discarding it, a later reconnect surfaces the+ # failure in manager errors, and the still-uncleaned server is never+ # started again as a second connection.+ class EmergencyCleanupFailingServer(TaskBoundServer):+ def __init__(self, mode: str) -> None:+ super().__init__()+ self.connect_started = asyncio.Event()+ self.mode = mode+ self.connect_calls = 0+ self.cleanup_calls = 0++ async def connect(self) -> None:+ self._connect_task = asyncio.current_task()+ self.connect_calls += 1+ self.connect_started.set()+ await asyncio.sleep(3600)++ async def cleanup(self) -> None:+ self.cleanup_calls += 1+ await super().cleanup()+ if self.mode == "raise":+ raise RuntimeError("emergency cleanup failed")+ await asyncio.sleep(3600)++ server = EmergencyCleanupFailingServer(mode)+ manager = MCPServerManager([server], connect_in_parallel=True, cleanup_timeout_seconds=0.2)++ connect_all = asyncio.create_task(manager.connect_all())+ await asyncio.wait_for(server.connect_started.wait(), timeout=TEST_TIMEOUT_SECONDS)+ manager._workers[server]._task.cancel()+ await asyncio.wait_for(connect_all, timeout=TEST_TIMEOUT_SECONDS)++ assert server.connect_calls == 1+ assert server.cleanup_calls == 1++ await manager.reconnect()++ assert server.connect_calls == 1+ assert manager.failed_servers == [server]+ assert manager.active_servers == []+ if mode == "raise":+ assert str(manager.errors[server]) == "emergency cleanup failed"+ else:+ assert isinstance(manager.errors[server], asyncio.TimeoutError)+ worker = manager._workers[server]+ assert worker.is_done+ assert worker.cleanup_error is not None+++@pytest.mark.asyncio+async def test_worker_task_is_created_through_the_loop_task_factory() -> None:+ # Worker creation must honor the event loop's task factory so applications+ # supervising or instrumenting tasks through set_task_factory can discover+ # the connection worker.+ loop = asyncio.get_running_loop()+ created: list[asyncio.Task[object]] = []++ def factory(+ loop_: asyncio.AbstractEventLoop, coro: Coroutine[Any, Any, object], **kwargs: object+ ) -> asyncio.Task[object]:+ task = asyncio.Task(coro, loop=loop_, **kwargs) # type: ignore[arg-type]+ created.append(task)+ return task++ loop.set_task_factory(factory)+ try:+ server = TaskBoundServer()+ manager = MCPServerManager([server], connect_in_parallel=True)+ await asyncio.wait_for(manager.connect_all(), timeout=TEST_TIMEOUT_SECONDS)+ worker = manager._workers[server]+ assert worker._task in created+ finally:+ loop.set_task_factory(None)