Backtestsopenai-agents-pythonrun-20261024-184640#5324

Issue #5324 Merged fix PR #5326 Run 24 Oct 2026 · run-20261024-184640 openai/openai-agents-python

Run-wide tool exception formatting: add tool_exception kind to RunConfig.tool_error_formatter

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches add `kind="tool_exception"` with an `error` field, apply the run formatter only on the default failure path, preserve `failure_error_function=None` re-raising, fall back to the default on formatter crash or non-string return, and pass all independent tests. The differences are in coverage and completeness. Patch A triggers the formatter whenever the tool's failure handler *is* `default_tool_error_function`, so SDK-built tools that pass the default explicitly (`Agent.as_tool`, MCP tools) also get the run-wide policy; it adds tests for both and wires the formatter into realtime sessions with a test. That matches the issue's motivation ("one run-wide policy" so tools added later don't silently fall back) and the stated rule that it applies "when a tool uses the default failure_error_function". Patch B keys off the existing `_use_default_failure_error_function` flag only, deliberately excludes as_tool/MCP tools and `MCPToolCancellationError`, and documents these gaps in a docstring; that leaves the issue's run-wide goal unmet for a common class of tools and adds exclusions the issue never asked for. B also ships no documentation changes, whereas A updates `docs/running_agents.md` and `docs/tools.md` where the formatter kinds are enumerated, and updates the `default_tool_error_function` docstring. On craft, A factors the logic into a clearly named helper with `_debug.DONT_LOG_TOOL_DATA`-aware logging; B inlines a nested block inside the existing function. B's test file is larger and includes useful redaction/namespace checks (and uses `qualified_tool_name`, a minor plus), but several tests exist to lock in B's own exclusions rather than the issue's behaviour, and the concurrency/cancellation test is tangential. Weighing correctness and scope against the issue's ask, A is the more complete and better-integrated fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

14m 20s

Wall-clock time in the container

Full test suite

No new failures

5,010 passed · 5 skipped · 5,015 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

440vs 603 merged

Changed lines, ours vs merged

Please read this first

  • Have you read the docs? (Tools, RunConfig)
  • Have you searched for related issues? (closest: #2658 / #3707, which added tool_error_formatter for other kinds)

Describe the feature

Since 0.23.0 (#5112), default_tool_error_function returns a fixed "An error occurred while running the tool. Please try again.". That is a sensible secure default. The only way to send approved error detail back to the model is a failure_error_function on each tool.

Apps with many tools usually want one run-wide policy instead. For example, "expose messages from our own ToolInputError, redact everything else". Today that means decorating every @function_tool call site. Tools added later silently fall back to the generic message, so the model can no longer self-correct bad arguments.

RunConfig.tool_error_formatter already exists for this kind of per-run policy, but ToolErrorFormatterArgs.kind only covers "approval_rejected" and "tool_not_found".

Proposal: add a kind="tool_exception" to tool_error_formatter, with the exception on the args (e.g. error: Exception | None). It would apply only when a tool uses the default failure_error_function, and returning None keeps the redacted default. Precedence would be: an explicit per-tool failure_error_function wins, then the run-level formatter, then the SDK default. Secure by default is preserved, and nothing changes unless the formatter opts in.

class ToolInputError(Exception): ...

def formatter(args: ToolErrorFormatterArgs) -> str | None:
    if args.kind == "tool_exception" and isinstance(args.error, ToolInputError):
        return f"Invalid input: {args.error}"
    return None  # keep SDK default

Runner.run(agent, input, run_config=RunConfig(tool_error_formatter=formatter))

Repro of current behaviour (0.23.1 vs 0.22.3)

A tool that raises ValueError("order_id must start with 'ORD-', got '123'"), run with a fake model that captures the next turn's input:

versionfunction_call_output sent to the model
0.22.3An error occurred while running the tool. Please try again. Error: order_id must start with 'ORD-', got '123'
0.23.1An error occurred while running the tool. Please try again.

On 0.23.1 the only fix is per tool (@function_tool(failure_error_function=...)). There is no run-wide equivalent.

Happy to help test a change if useful.

The two patches

Our patch+432 −8
docs/running_agents.mdindex 8d10f9c7..f65580d6 100644@@ -162,7 +162,7 @@ Use `RunConfig` to override behavior for a single run without changing each agen -   [`tool_execution`][agents.run.RunConfig.tool_execution]: Configure SDK-side execution behavior for local tool calls, such as limiting how many local function tool calls run at once. -   [`tool_not_found_behavior`][agents.run.RunConfig.tool_not_found_behavior]: Configure how the runner handles model-emitted function tool calls whose tool name does not match any function tool available to the current agent. The default raises `ModelBehaviorError`; opt in to return a model-visible error output instead. -   [`tool_name_collision_policy`][agents.run.RunConfig.tool_name_collision_policy]: Configure how the runner handles unnamespaced function-tool and handoff names that collide. The default, `"warn"`, logs an actionable warning and exposes only the current dispatch winner; `"error"` raises `UserError` before the model is called. Strict validation for namespaced and deferred-loading tools is unchanged.--   [`tool_error_formatter`][agents.run.RunConfig.tool_error_formatter]: Customize model-visible tool error messages, such as approval rejections and opt-in tool-not-found outputs.+-   [`tool_error_formatter`][agents.run.RunConfig.tool_error_formatter]: Customize model-visible tool error messages, such as approval rejections, opt-in tool-not-found outputs, and default tool exception messages.  Nested handoffs are available as an opt-in beta. Enable ordered transcript compaction by passing `RunConfig(nest_handoff_history=True)` or set `handoff(..., nest_handoff_history=True)` to turn it on for a specific handoff. The built-in mapper places generated assistant summary segments around lossless message items instead of collapsing the whole transcript into one message. If you prefer to keep the raw transcript (the default), leave the flag unset or provide a `handoff_input_filter` (or `handoff_history_mapper`) that forwards the conversation exactly as you need. To change the wrapper text used in generated summary segments without writing a custom mapper, call [`set_conversation_history_wrappers`][agents.handoffs.set_conversation_history_wrappers] (and [`reset_conversation_history_wrappers`][agents.handoffs.reset_conversation_history_wrappers] to restore the defaults). @@ -221,15 +221,35 @@ Use `tool_error_formatter` to customize the message that is returned to the mode  The formatter receives [`ToolErrorFormatterArgs`][agents.run_config.ToolErrorFormatterArgs] with: --   `kind`: The error category, such as `"approval_rejected"` or `"tool_not_found"`.+-   `kind`: The error category: `"approval_rejected"`, `"tool_not_found"`, or `"tool_exception"`. -   `tool_type`: The tool runtime (`"function"`, `"computer"`, `"shell"`, `"apply_patch"`, or `"custom"`). -   `tool_name`: The tool name. -   `call_id`: The tool call ID. -   `default_message`: The SDK's default model-visible message. -   `run_context`: The active run context wrapper.+-   `error`: The exception raised by the tool for `"tool_exception"`, otherwise `None`.  Return a string to replace the message, or `None` to use the SDK default. +`"tool_exception"` applies when a function tool raises and would otherwise return the SDK default error message. A tool's own `failure_error_function` takes precedence, and `failure_error_function=None` still re-raises. This lets you expose approved error details run-wide while keeping everything else redacted:++```python+from agents import RunConfig, ToolErrorFormatterArgs+++class ToolInputError(Exception):+    pass+++def format_tool_errors(args: ToolErrorFormatterArgs[None]) -> str | None:+    if args.kind == "tool_exception" and isinstance(args.error, ToolInputError):+        return f"Invalid input: {args.error}"+    return None+++run_config = RunConfig(tool_error_formatter=format_tool_errors)+```+ ```python from agents import Agent, RunConfig, Runner, ToolErrorFormatterArgs docs/tools.mdindex 47c301d6..89f96ec0 100644@@ -645,7 +645,7 @@ except ToolTimeoutError as e:  When you create a function tool via `@function_tool`, you can pass a `failure_error_function`. This is a function that provides an error response to the LLM in case the tool call crashes. --   By default (i.e. if you don't pass anything), it runs a `default_tool_error_function` which tells the LLM an error occurred.+-   By default (i.e. if you don't pass anything), it runs a `default_tool_error_function` which tells the LLM an error occurred. To expose approved error details for every such tool in a run, set [`RunConfig.tool_error_formatter`][agents.run.RunConfig.tool_error_formatter] and handle `kind="tool_exception"`. -   If you pass your own error function, it runs that instead, and sends the response to the LLM. -   If you explicitly pass `None`, then any tool call errors will be re-raised for you to handle. This could be a `ModelBehaviorError` if the model produced invalid JSON, or a `UserError` if your code crashed, etc. src/agents/realtime/session.pyindex d9c502d8..b42cf34e 100644@@ -1222,6 +1222,7 @@ class RealtimeSession(RealtimeModelListener):                     tool_arguments=event.arguments,                     agent=agent,                 )+                tool_context._tool_error_formatter = self._run_config.get("tool_error_formatter")                 approval_status = await self._maybe_request_tool_approval(                     event,                     function_tool=func_tool,src/agents/run_config.pyindex ac013cec..3af43481 100644@@ -83,8 +83,12 @@ ToolNameCollisionPolicy = Literal["warn", "error"] class ToolErrorFormatterArgs(Generic[TContext]):     """Data passed to ``RunConfig.tool_error_formatter`` callbacks.""" -    kind: Literal["approval_rejected", "tool_not_found"]-    """The category of tool error being formatted."""+    kind: Literal["approval_rejected", "tool_not_found", "tool_exception"]+    """The category of tool error being formatted.++    ``"tool_exception"`` is used when a function tool raises and would otherwise return the SDK+    default error message (i.e. the tool does not set its own ``failure_error_function``).+    """      tool_type: Literal["function", "computer", "shell", "apply_patch", "custom"]     """The tool runtime that produced the error."""@@ -101,6 +105,14 @@ class ToolErrorFormatterArgs(Generic[TContext]):     run_context: RunContextWrapper[TContext]     """The active run context for the current execution.""" +    error: Exception | None = None+    """The exception raised by the tool for ``kind="tool_exception"``; ``None`` otherwise.++    This is the same exception a per-tool ``failure_error_function`` would receive, so it may be+    an SDK wrapper (for example ``ModelBehaviorError`` for invalid arguments, ``UserError`` for+    MCP tool failures, or a cancellation error) rather than the original tool exception.+    """+  ToolErrorFormatter = Callable[[ToolErrorFormatterArgs[Any]], MaybeAwaitable[str | None]] @@ -458,6 +470,8 @@ class RunConfig:     tool_error_formatter: ToolErrorFormatter | None = None     """Optional callback that formats tool error messages returned to the model. +    For ``kind="tool_exception"`` it only applies to tools that use the default+    ``failure_error_function``; an explicit per-tool ``failure_error_function`` takes precedence.     Returning ``None`` falls back to the SDK default message.     """ src/agents/tool.pyindex 63ec2aa1..fa64c392 100644@@ -75,7 +75,7 @@ from .computer import AsyncComputer, Computer from .editor import ApplyPatchEditor, ApplyPatchOperation from .exceptions import ModelBehaviorError, ToolTimeoutError, UserError from .function_schema import DocstringStyle, function_schema, generate_func_documentation-from .logger import log_tool_action_warning, logger+from .logger import log_tool_action_error, log_tool_action_warning, logger from .run_context import RunContextWrapper from .strict_schema import _copy_json_schema, ensure_strict_json_schema from .tool_context import ToolContext@@ -90,6 +90,7 @@ if TYPE_CHECKING:      from .agent import Agent, AgentBase     from .items import RunItem, ToolApprovalItem+    from .run_config import ToolErrorFormatter   ToolParams = ParamSpec("ToolParams")@@ -1980,7 +1981,8 @@ def _log_function_tool_invocation(*, tool_name: str, input_json: str) -> None: def default_tool_error_function(ctx: RunContextWrapper[Any], error: Exception) -> str:     """Return a fixed error response without exposing the exception to the model. -    Provide a custom ``failure_error_function`` to return application-approved error details.+    Provide a custom ``failure_error_function`` to return application-approved error details,+    or set ``RunConfig.tool_error_formatter`` to handle ``kind="tool_exception"`` run-wide.     """     return "An error occurred while running the tool. Please try again." @@ -2089,11 +2091,78 @@ async def maybe_invoke_function_tool_failure_error_function(     result = failure_error_function(context, formatter_error)     if inspect.isawaitable(result):         result = await result+    if failure_error_function is default_tool_error_function and isinstance(result, str):+        result = await _maybe_format_tool_exception_with_run_formatter(+            function_tool=function_tool,+            context=context,+            error=formatter_error,+            default_message=result,+        )     if function_tool._use_default_failure_error_function and isinstance(context, ToolContext):         setattr(context, _DEFAULT_FAILURE_HANDLED_ATTR, True)     return result  +def _resolve_run_tool_error_formatter(+    context: RunContextWrapper[Any],+) -> ToolErrorFormatter | None:+    """Return the run-level ``tool_error_formatter`` reachable from a tool context, if any."""+    if not isinstance(context, ToolContext):+        return None+    if context.run_config is not None:+        return context.run_config.tool_error_formatter+    return context._tool_error_formatter+++async def _maybe_format_tool_exception_with_run_formatter(+    *,+    function_tool: FunctionTool,+    context: RunContextWrapper[Any],+    error: Exception,+    default_message: str,+) -> str:+    """Let ``RunConfig.tool_error_formatter`` replace the SDK default tool exception message."""+    formatter = _resolve_run_tool_error_formatter(context)+    if formatter is None:+        return default_message++    from .run_config import ToolErrorFormatterArgs++    assert isinstance(context, ToolContext)+    try:+        maybe_message = formatter(+            ToolErrorFormatterArgs(+                kind="tool_exception",+                tool_type="function",+                tool_name=function_tool.name,+                call_id=context.tool_call_id,+                default_message=default_message,+                run_context=context,+                error=error,+            )+        )+        message = await maybe_message if inspect.isawaitable(maybe_message) else maybe_message+    except Exception as exc:+        log_tool_action_error(logger, "Tool error formatter failed for tool exception", exc)+        return default_message++    if message is None:+        return default_message++    if not isinstance(message, str):+        if _debug.DONT_LOG_TOOL_DATA:+            logger.error("Tool error formatter returned a non-string value for a tool exception")+        else:+            logger.error(+                "Tool error formatter returned non-string for tool exception in %s: %s",+                function_tool.name,+                type(message).__name__,+            )+        return default_message++    return message++ def _annotation_expr_name(expr: ast.expr) -> str | None:     """Return the unqualified type name for a string annotation expression node."""     if isinstance(expr, ast.Name):src/agents/tool_context.pyindex 89ecc930..50911fda 100644@@ -20,7 +20,7 @@ if TYPE_CHECKING:     from ._function_tool_arguments import PreparedFunctionArguments     from .agent import AgentBase     from .items import ToolApprovalItem, TResponseInputItem-    from .run_config import RunConfig+    from .run_config import RunConfig, ToolErrorFormatter     from .run_context import _ApprovalKey, _ApprovalRecord  @@ -118,6 +118,8 @@ class ToolContext(RunContextWrapper[TContext]):         # Internal adapter hook used to attach SDK-only custom data to the emitted output item.         self._custom_data: dict[str, Any] | None = None         self._function_tool_arguments: PreparedFunctionArguments | None = None+        # Run-level tool error formatter for runtimes without a ``RunConfig`` (e.g. realtime).+        self._tool_error_formatter: ToolErrorFormatter | None = None      @property     def qualified_tool_name(self) -> str:tests/mcp/test_runner_calls_mcp.pyindex 8ba9d1a8..e2a23690 100644@@ -8,8 +8,10 @@ from agents import (     Agent,     FunctionTool,     ModelBehaviorError,+    RunConfig,     RunContextWrapper,     Runner,+    ToolErrorFormatterArgs,     ToolGuardrailFunctionOutput,     ToolInputGuardrailData,     ToolOutputGuardrailData,@@ -484,3 +486,36 @@ async def test_runner_emits_mcp_error_tool_call_output_item(streaming: bool):         wrapped_error,     )     assert tool_output_items[0].output == expected_error_message+++@pytest.mark.asyncio+async def test_runner_mcp_tool_failure_uses_run_tool_error_formatter():+    """MCP tools use the SDK default failure formatter, so the run-level formatter applies."""+    server = CrashingFakeMCPServer()+    server.add_tool("crashing_tool", {})++    model = ScriptedModel()+    agent = Agent(name="test", model=model, mcp_servers=[server])+    model.extend(+        [+            [get_function_tool_call("crashing_tool", "{}")],+            [get_text_message("done")],+        ]+    )+    seen: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        seen.append(args)+        return f"{args.kind}: {args.error}"++    result = await Runner.run(+        agent, input="user_message", run_config=RunConfig(tool_error_formatter=formatter)+    )++    assert result.final_output == "done"+    assert [args.kind for args in seen] == ["tool_exception"]+    assert seen[0].tool_name == "crashing_tool"+    assert _model_tool_outputs(model) == [+        "tool_exception: Error invoking MCP tool crashing_tool on server "+        "'fake_mcp_server': Crash!"+    ]tests/realtime/test_session_tool_error_events.pyindex 94eba246..c680f40c 100644@@ -2,6 +2,7 @@  import asyncio import logging+from typing import Any  import pytest @@ -12,6 +13,7 @@ from agents.realtime.events import RealtimeError, RealtimeSessionEvent, Realtime from agents.realtime.model_events import RealtimeModelToolCallEvent from agents.realtime.model_inputs import RealtimeModelSendToolOutput from agents.realtime.testing import RealtimeStep, ScriptedRealtimeModel+from agents.run_config import ToolErrorFormatterArgs   class UnprintableToolError(RuntimeError):@@ -166,3 +168,51 @@ async def test_async_tool_failure_events_are_safe(         finally:             consumer.cancel()             await asyncio.gather(consumer, return_exceptions=True)+++@pytest.mark.parametrize("async_tool_calls", [False, True])+@pytest.mark.asyncio+async def test_tool_exception_uses_realtime_tool_error_formatter(async_tool_calls: bool) -> None:+    seen_kinds: list[str] = []++    @tool+    async def lookup(value: str) -> str:+        raise ValueError(f"bad value {value}")++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        seen_kinds.append(args.kind)+        return f"{args.tool_name} ({args.call_id}): {args.error}"++    model = ScriptedRealtimeModel(steps=[RealtimeStep(expect=RealtimeModelSendToolOutput)])+    tool_end = asyncio.Event()++    async def consume(session: RealtimeSession) -> None:+        async for event in session:+            if isinstance(event, RealtimeToolEnd):+                tool_end.set()++    async with RealtimeSession(+        model,+        RealtimeAgent(name="test", tools=[lookup]),+        None,+        run_config={"tool_error_formatter": formatter, "async_tool_calls": async_tool_calls},+    ) as session:+        consumer = asyncio.create_task(consume(session))+        try:+            await model.emit(+                RealtimeModelToolCallEvent(+                    name="lookup", call_id="call_1", arguments='{"value":"x"}'+                )+            )+            await asyncio.wait_for(tool_end.wait(), timeout=2)+        finally:+            consumer.cancel()+            await asyncio.gather(consumer, return_exceptions=True)++    assert seen_kinds == ["tool_exception"]+    sent_outputs = [+        event.output+        for event in model.sent_events+        if isinstance(event, RealtimeModelSendToolOutput)+    ]+    assert sent_outputs == ["lookup (call_1): bad value x"]tests/test_tool_exception_formatter.pynew file mode 100644index 00000000..510e315a@@ -0,0 +1,233 @@+"""Run-level ``tool_error_formatter`` handling for ``kind="tool_exception"``."""++from __future__ import annotations++from typing import Any++import pytest++from agents import (+    Agent,+    RunConfig,+    Runner,+    ToolErrorFormatterArgs,+    UserError,+    default_tool_error_function,+    function_tool,+)+from agents.testing import ScriptedModel++from .test_responses import get_function_tool_call, get_text_message++DEFAULT_MESSAGE = "An error occurred while running the tool. Please try again."+++class ToolInputError(Exception):+    pass+++def _tool_outputs(model: ScriptedModel) -> list[Any]:+    second_turn_input = model.calls[-1].input+    assert isinstance(second_turn_input, list)+    return [+        item.get("output")+        for item in second_turn_input+        if isinstance(item, dict) and item.get("type") == "function_call_output"+    ]+++async def _run_failing_tool(+    *,+    formatter: Any,+    error: Exception,+    streaming: bool = False,+    **tool_kwargs: Any,+) -> list[Any]:+    def lookup_order(order_id: str) -> str:+        raise error++    tool = function_tool(lookup_order, **tool_kwargs)+    model = ScriptedModel()+    agent = Agent(name="test", model=model, tools=[tool])+    model.extend(+        [+            [get_function_tool_call("lookup_order", '{"order_id": "123"}', call_id="call_1")],+            [get_text_message("done")],+        ]+    )+    run_config = RunConfig(tool_error_formatter=formatter)+    if streaming:+        streamed = Runner.run_streamed(agent, input="start", run_config=run_config)+        async for _ in streamed.stream_events():+            pass+        assert streamed.final_output == "done"+    else:+        result = await Runner.run(agent, input="start", run_config=run_config)+        assert result.final_output == "done"+    return _tool_outputs(model)+++@pytest.mark.asyncio+@pytest.mark.parametrize("streaming", [False, True])+async def test_tool_exception_formatter_exposes_approved_error(streaming: bool) -> None:+    seen: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        seen.append(args)+        if args.kind == "tool_exception" and isinstance(args.error, ToolInputError):+            return f"Invalid input: {args.error}"+        return None++    outputs = await _run_failing_tool(+        formatter=formatter,+        error=ToolInputError("order_id must start with 'ORD-', got '123'"),+        streaming=streaming,+    )++    assert outputs == ["Invalid input: order_id must start with 'ORD-', got '123'"]+    assert len(seen) == 1+    args = seen[0]+    assert args.kind == "tool_exception"+    assert args.tool_type == "function"+    assert args.tool_name == "lookup_order"+    assert args.call_id == "call_1"+    assert args.default_message == DEFAULT_MESSAGE+++@pytest.mark.asyncio+async def test_tool_exception_formatter_returning_none_keeps_redacted_default() -> None:+    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        if args.kind == "tool_exception" and isinstance(args.error, ToolInputError):+            return f"Invalid input: {args.error}"+        return None++    outputs = await _run_failing_tool(formatter=formatter, error=ValueError("secret detail"))++    assert outputs == [DEFAULT_MESSAGE]+++@pytest.mark.asyncio+async def test_tool_exception_async_formatter_is_awaited() -> None:+    async def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        return f"async: {args.error}"++    outputs = await _run_failing_tool(formatter=formatter, error=ValueError("bad"))++    assert outputs == ["async: bad"]+++@pytest.mark.asyncio+async def test_explicit_failure_error_function_wins_over_run_formatter() -> None:+    kinds: list[str] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        kinds.append(args.kind)+        return "from run formatter"++    outputs = await _run_failing_tool(+        formatter=formatter,+        error=ValueError("bad"),+        failure_error_function=lambda ctx, err: f"per-tool: {err}",+    )++    assert outputs == ["per-tool: bad"]+    assert kinds == []+++@pytest.mark.asyncio+async def test_explicit_default_tool_error_function_uses_run_formatter() -> None:+    """Wrappers such as ``Agent.as_tool`` and MCP pass the SDK default explicitly."""++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        return f"formatted: {args.error}"++    outputs = await _run_failing_tool(+        formatter=formatter,+        error=ValueError("bad"),+        failure_error_function=default_tool_error_function,+    )++    assert outputs == ["formatted: bad"]+++@pytest.mark.asyncio+async def test_agent_as_tool_failure_uses_run_formatter() -> None:+    inner_model = ScriptedModel()+    inner_model.extend([ToolInputError("inner agent failed")])+    inner_agent = Agent(name="inner", model=inner_model)++    model = ScriptedModel()+    agent = Agent(+        name="outer",+        model=model,+        tools=[inner_agent.as_tool(tool_name="inner_tool", tool_description="inner")],+    )+    model.extend(+        [+            [get_function_tool_call("inner_tool", '{"input": "hi"}', call_id="call_inner")],+            [get_text_message("done")],+        ]+    )+    seen_errors: list[Exception | None] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        if args.kind != "tool_exception":+            return None+        seen_errors.append(args.error)+        return f"{args.tool_name} failed: {args.error}"++    result = await Runner.run(+        agent, input="start", run_config=RunConfig(tool_error_formatter=formatter)+    )++    assert result.final_output == "done"+    assert _tool_outputs(model) == ["inner_tool failed: inner agent failed"]+    assert len(seen_errors) == 1+    assert isinstance(seen_errors[0], ToolInputError)+++@pytest.mark.asyncio+async def test_failure_error_function_none_still_raises_with_run_formatter() -> None:+    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        return "should not be used"++    def lookup_order(order_id: str) -> str:+        raise ToolInputError("boom")++    model = ScriptedModel()+    agent = Agent(+        name="test",+        model=model,+        tools=[function_tool(lookup_order, failure_error_function=None)],+    )+    model.extend([[get_function_tool_call("lookup_order", '{"order_id": "1"}')]])++    with pytest.raises(UserError, match="Error running tool lookup_order: boom"):+        await Runner.run(agent, input="start", run_config=RunConfig(tool_error_formatter=formatter))+++@pytest.mark.asyncio+async def test_tool_exception_formatter_failure_falls_back_to_default() -> None:+    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        raise RuntimeError("formatter crashed")++    outputs = await _run_failing_tool(formatter=formatter, error=ValueError("bad"))++    assert outputs == [DEFAULT_MESSAGE]+++@pytest.mark.asyncio+async def test_tool_exception_formatter_non_string_falls_back_to_default() -> None:+    def formatter(args: ToolErrorFormatterArgs[Any]) -> Any:+        return 123++    outputs = await _run_failing_tool(formatter=formatter, error=ValueError("bad"))++    assert outputs == [DEFAULT_MESSAGE]+++@pytest.mark.asyncio+async def test_tool_exception_without_formatter_keeps_default() -> None:+    outputs = await _run_failing_tool(formatter=None, error=ValueError("secret detail"))++    assert outputs == [DEFAULT_MESSAGE] 
Merged fix · PR #5326+601 −2
src/agents/run_config.pyindex ac013cec0..9fba7e843 100644@@ -83,7 +83,7 @@ ToolNameCollisionPolicy = Literal["warn", "error"] class ToolErrorFormatterArgs(Generic[TContext]):     """Data passed to ``RunConfig.tool_error_formatter`` callbacks.""" -    kind: Literal["approval_rejected", "tool_not_found"]+    kind: Literal["approval_rejected", "tool_not_found", "tool_exception"]     """The category of tool error being formatted."""      tool_type: Literal["function", "computer", "shell", "apply_patch", "custom"]@@ -101,6 +101,13 @@ class ToolErrorFormatterArgs(Generic[TContext]):     run_context: RunContextWrapper[TContext]     """The active run context for the current execution.""" +    error: Exception | None = None+    """The original exception for ``tool_exception``; otherwise ``None``.++    This exception can contain sensitive data. Return only application-approved details;+    the SDK does not include the original exception in handled-error traces for this policy.+    """+  ToolErrorFormatter = Callable[[ToolErrorFormatterArgs[Any]], MaybeAwaitable[str | None]] @@ -459,6 +466,15 @@ class RunConfig:     """Optional callback that formats tool error messages returned to the model.      Returning ``None`` falls back to the SDK default message.++    For ordinary function-tool exceptions, this callback applies only when the tool uses+    the implicit default failure handler. Explicit per-tool handlers, including ``None``+    (propagate errors), take precedence. Tools created by ``Agent.as_tool()`` and local MCP+    factories retain their existing ``failure_error_function`` policies; this callback+    does not format their exceptions. Timeouts, cancellation, and schema-backed+    program-call failures retain their existing policies. A callback failure falls back+    to the default message. Returned messages may enter model input, history, and traces;+    return only application-approved details.     """      session_settings: SessionSettings | None = Nonesrc/agents/tool.pyindex 63ec2aa16..6f6e91d17 100644@@ -73,7 +73,7 @@ from ._tool_identity import ( ) from .computer import AsyncComputer, Computer from .editor import ApplyPatchEditor, ApplyPatchOperation-from .exceptions import ModelBehaviorError, ToolTimeoutError, UserError+from .exceptions import MCPToolCancellationError, ModelBehaviorError, ToolTimeoutError, UserError from .function_schema import DocstringStyle, function_schema, generate_func_documentation from .logger import log_tool_action_warning, logger from .run_context import RunContextWrapper@@ -2091,6 +2091,36 @@ async def maybe_invoke_function_tool_failure_error_function(         result = await result     if function_tool._use_default_failure_error_function and isinstance(context, ToolContext):         setattr(context, _DEFAULT_FAILURE_HANDLED_ATTR, True)+        if (+            isinstance(error, Exception)+            and not isinstance(error, MCPToolCancellationError)+            and context.run_config is not None+            and context.run_config.tool_error_formatter is not None+        ):+            # Keep the default failure policy identity: approving model feedback must not+            # opt the original exception into handled-error traces.+            from .run_config import ToolErrorFormatterArgs++            try:+                message = context.run_config.tool_error_formatter(+                    ToolErrorFormatterArgs(+                        kind="tool_exception",+                        tool_type="function",+                        tool_name=context.qualified_tool_name,+                        call_id=context.tool_call_id,+                        default_message=result,+                        run_context=context,+                        error=error,+                    )+                )+                message = await message if inspect.isawaitable(message) else message+            except Exception as exc:+                log_tool_action_warning(logger, "Tool error formatter failed", exc)+            else:+                if isinstance(message, str):+                    return message+                if message is not None:+                    logger.warning("Tool error formatter returned a non-string value")     return result  tests/test_tool_exception_formatter.pynew file mode 100644index 000000000..02da8a9b6@@ -0,0 +1,553 @@+from __future__ import annotations++import asyncio+import json+import logging+from typing import Any++import pytest++from agents import Agent, RunConfig, RunContextWrapper, Runner, ToolErrorFormatterArgs, UserError+from agents.decorators import tool+from agents.testing import ScriptedModel+from agents.tool import default_tool_error_function++from .test_responses import get_function_tool_call, get_text_message+from .testing_processor import SPAN_PROCESSOR_TESTING++SECRET = "synthetic-private-exception-detail"+GENERIC = "An error occurred while running the tool. Please try again."+APPROVED = "Order IDs must start with ORD-."+++def make_model(name: str) -> ScriptedModel:+    return ScriptedModel(+        [[get_function_tool_call(name, "{}", call_id="call-5324")], [get_text_message("done")]]+    )+++def model_output(model: ScriptedModel) -> Any:+    items = model.calls[-1].input+    assert isinstance(items, list)+    return next(item["output"] for item in items if item.get("type") == "function_call_output")+++def assert_redacted(caplog: pytest.LogCaptureFixture, result: Any, model: ScriptedModel) -> None:+    assert SECRET not in json.dumps(model.calls[-1].input)+    assert SECRET not in json.dumps(result.to_input_list())+    spans = [span.export() for span in SPAN_PROCESSOR_TESTING.get_ordered_spans()]+    assert spans+    assert SECRET not in json.dumps(spans)+    for record in caplog.records:+        assert SECRET not in repr(record.__dict__)+        assert record.exc_info is None+        assert record.exc_text is None+        assert SECRET not in logging.Formatter().format(record)+++@pytest.mark.asyncio+@pytest.mark.parametrize("streamed", [False, True])+@pytest.mark.parametrize("async_handler", [False, True])+async def test_run_formatter_returns_only_approved_feedback(+    streamed: bool, async_handler: bool, caplog: pytest.LogCaptureFixture+) -> None:+    class ToolInputError(Exception):+        pass++    def fail() -> str:+        try:+            raise RuntimeError(f"cause-{SECRET}")+        except RuntimeError as cause:+            raise ToolInputError(SECRET) from cause++    async def fail_async() -> str:+        return fail()++    failing_tool = tool(fail_async if async_handler else fail)+    received: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        received.append(args)+        assert isinstance(args.error, ToolInputError)+        return APPROVED++    async def async_formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        await asyncio.sleep(0)+        return formatter(args)++    config = RunConfig(+        tool_error_formatter=async_formatter if async_handler else formatter,+        trace_include_sensitive_data=True,+    )+    model = make_model(failing_tool.name)+    agent = Agent(name="test", tools=[failing_tool], model=model)+    with caplog.at_level(logging.DEBUG, logger="openai.agents"):+        if streamed:+            result = Runner.run_streamed(agent, "start", run_config=config)+            async for _ in result.stream_events():+                pass+        else:+            result = await Runner.run(agent, "start", run_config=config)+    assert result.final_output == "done"+    assert model_output(model) == APPROVED+    assert len(received) == 1+    args = received[0]+    assert (args.kind, args.tool_type, args.tool_name, args.call_id) == (+        "tool_exception",+        "function",+        failing_tool.name,+        "call-5324",+    )+    assert args.default_message == GENERIC+    assert_redacted(caplog, result, model)+++@pytest.mark.asyncio+@pytest.mark.parametrize("response", ["none", "raise", "invalid", "empty"])+async def test_formatter_fallback_does_not_expose_exception_or_bad_return(+    response: str, caplog: pytest.LogCaptureFixture+) -> None:+    class UnprintableError(Exception):+        def __str__(self) -> str:+            raise AssertionError("must not stringify private exception")++    class UnprintableResult:+        def __str__(self) -> str:+            raise AssertionError("must not stringify invalid result")++        def __repr__(self) -> str:+            raise AssertionError("must not repr invalid result")++    @tool+    async def fail() -> str:+        raise UnprintableError(SECRET)++    async def formatter(args: ToolErrorFormatterArgs[Any]) -> Any:+        await asyncio.sleep(0)+        if response == "raise":+            raise UnprintableError(SECRET) from args.error+        if response == "invalid":+            return UnprintableResult()+        if response == "empty":+            return ""+        return None++    model = make_model(fail.name)+    with caplog.at_level(logging.DEBUG, logger="openai.agents"):+        result = await Runner.run(+            Agent(name="test", tools=[fail], model=model),+            "start",+            run_config=RunConfig(tool_error_formatter=formatter, trace_include_sensitive_data=True),+        )+    assert result.final_output == "done"+    assert model_output(model) == ("" if response == "empty" else GENERIC)+    assert_redacted(caplog, result, model)+++@pytest.mark.asyncio+@pytest.mark.parametrize("policy", ["custom", "propagate", "explicit-default"])+async def test_explicit_tool_policy_wins_over_run_formatter(policy: str) -> None:+    def per_tool(ctx: RunContextWrapper[Any], error: Exception) -> str:+        return "per-tool message"++    @tool(+        failure_error_function=(+            per_tool+            if policy == "custom"+            else default_tool_error_function+            if policy == "explicit-default"+            else None+        )+    )+    async def fail() -> str:+        raise ValueError(SECRET)++    called: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        called.append(args)+        return APPROVED++    model = make_model(fail.name)+    agent = Agent(name="test", tools=[fail], model=model)+    config = RunConfig(tool_error_formatter=formatter)+    if policy == "propagate":+        with pytest.raises(UserError, match=SECRET):+            await Runner.run(agent, "start", run_config=config)+    else:+        await Runner.run(agent, "start", run_config=config)+        assert model_output(model) == ("per-tool message" if policy == "custom" else GENERIC)+    assert not called+++@pytest.mark.asyncio+@pytest.mark.parametrize("failure", ["cancel", "timeout"])+async def test_timeout_and_tool_cancellation_keep_existing_policies(failure: str) -> None:+    @tool(timeout=0.01 if failure == "timeout" else None)+    async def fail() -> str:+        if failure == "cancel":+            raise asyncio.CancelledError(SECRET)+        await asyncio.Event().wait()+        return "unreachable"++    called: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        called.append(args)+        return APPROVED++    model = make_model(fail.name)+    await Runner.run(+        Agent(name="test", tools=[fail], model=model),+        "start",+        run_config=RunConfig(tool_error_formatter=formatter),+    )+    assert not called+    assert model_output(model) == (+        "Tool 'fail' timed out after 0.01 seconds." if failure == "timeout" else GENERIC+    )+++@pytest.mark.asyncio+async def test_concurrent_runs_keep_their_own_formatter_and_parent_cancellation() -> None:+    @tool+    async def fail() -> str:+        raise ValueError(SECRET)++    entered = asyncio.Event()+    release = asyncio.Event()++    async def waiting_formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        entered.set()+        await release.wait()+        return "first run"++    first_model = make_model(fail.name)+    second_model = make_model(fail.name)+    first = asyncio.create_task(+        Runner.run(+            Agent(name="first", tools=[fail], model=first_model),+            "start",+            run_config=RunConfig(tool_error_formatter=waiting_formatter),+        )+    )+    try:+        await asyncio.wait_for(entered.wait(), timeout=5)+        second = await Runner.run(+            Agent(name="second", tools=[fail], model=second_model),+            "start",+            run_config=RunConfig(tool_error_formatter=lambda args: "second run"),+        )+        assert second.final_output == "done"+        assert model_output(second_model) == "second run"+        first.cancel()+        with pytest.raises(asyncio.CancelledError):+            await first+        assert len(first_model.calls) == 1+    finally:+        release.set()+        first.cancel()+        await asyncio.gather(first, return_exceptions=True)+++def test_formatter_args_preserve_old_positional_constructor() -> None:+    context = RunContextWrapper(None)+    args = ToolErrorFormatterArgs(+        "tool_not_found", "function", "missing", "call", "default", context+    )+    assert args.run_context is context+    assert args.error is None+++@pytest.mark.asyncio+@pytest.mark.parametrize("tool_redacted", [False, True])+async def test_formatter_failure_respects_local_diagnostic_opt_in(+    tool_redacted: bool, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture+) -> None:+    import agents._debug as debug++    monkeypatch.setattr(debug, "DONT_LOG_TOOL_DATA", tool_redacted)++    @tool+    async def fail() -> str:+        raise ValueError(SECRET)++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        raise RuntimeError(f"formatter-{SECRET}") from args.error++    model = make_model(fail.name)+    with caplog.at_level(logging.DEBUG, logger="openai.agents"):+        result = await Runner.run(+            Agent(name="test", tools=[fail], model=model),+            "start",+            run_config=RunConfig(tool_error_formatter=formatter, trace_include_sensitive_data=True),+        )+    assert model_output(model) == GENERIC+    assert SECRET not in json.dumps(result.to_input_list())+    assert SECRET not in json.dumps(+        [span.export() for span in SPAN_PROCESSOR_TESTING.get_ordered_spans()]+    )+    if tool_redacted:+        assert_redacted(caplog, result, model)+    else:+        assert SECRET in caplog.text+        assert any(record.exc_info is not None for record in caplog.records)+++@pytest.mark.asyncio+@pytest.mark.parametrize("program_call", [False, True])+async def test_schema_backed_tool_retains_program_failure_policy(program_call: bool) -> None:+    from openai.types.responses.response_function_tool_call import CallerProgram+    from openai.types.responses.response_output_item import Program++    from agents import ProgrammaticToolCallingTool++    @tool(+        allowed_callers=["direct", "programmatic"],+        output_json_schema={+            "type": "object",+            "properties": {"ok": {"type": "boolean"}},+            "required": ["ok"],+            "additionalProperties": False,+        },+    )+    async def fail() -> str:+        raise ValueError(SECRET)++    called: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        called.append(args)+        return APPROVED++    call = get_function_tool_call(fail.name, "{}")+    if program_call:+        call.caller = CallerProgram(type="program", caller_id="synthetic-program")+    program = Program(+        id="program-item",+        call_id="synthetic-program",+        code="fail()",+        fingerprint="synthetic-fingerprint",+        type="program",+    )+    model = ScriptedModel([[program, call] if program_call else [call], [get_text_message("done")]])+    agent = Agent(name="test", tools=[fail, ProgrammaticToolCallingTool()], model=model)+    config = RunConfig(tool_error_formatter=formatter)+    if program_call:+        with pytest.raises(UserError, match=SECRET):+            await Runner.run(agent, "start", run_config=config)+        assert not called+    else:+        await Runner.run(agent, "start", run_config=config)+        assert len(called) == 1+        assert model_output(model) == APPROVED+++@pytest.mark.asyncio+@pytest.mark.parametrize("policy", ["omitted", "custom", "explicit-default", "propagate"])+async def test_agent_tool_retains_factory_failure_policy(+    policy: str, caplog: pytest.LogCaptureFixture+) -> None:+    async def extract(result: Any) -> str:+        raise ValueError(SECRET)++    options: dict[str, Any] = {}+    if policy != "omitted":+        options["failure_error_function"] = (+            (lambda ctx, error: "per-tool")+            if policy == "custom"+            else default_tool_error_function+            if policy == "explicit-default"+            else None+        )+    child = Agent(name="child", model=ScriptedModel([[get_text_message("child done")]]))+    child_tool = child.as_tool("child", "Run child", custom_output_extractor=extract, **options)+    model = ScriptedModel(+        [[get_function_tool_call("child", '{"input":"task"}')], [get_text_message("done")]]+    )+    called: list[str] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        called.append(args.tool_name)+        return APPROVED++    agent = Agent(name="parent", model=model, tools=[child_tool])+    config = RunConfig(tool_error_formatter=formatter, trace_include_sensitive_data=True)+    if policy == "propagate":+        with pytest.raises(UserError, match=SECRET):+            await Runner.run(agent, "start", run_config=config)+    else:+        with caplog.at_level(logging.DEBUG, logger="openai.agents"):+            result = await Runner.run(agent, "start", run_config=config)+        assert (+            model_output(model)+            == {+                "omitted": GENERIC,+                "custom": "per-tool",+                "explicit-default": GENERIC,+            }[policy]+        )+        if policy == "omitted":+            assert_redacted(caplog, result, model)+    assert not called+++@pytest.mark.asyncio+@pytest.mark.parametrize(+    "level,policy",+    [+        ("agent", "omitted"),+        ("agent", "custom"),+        ("agent", "explicit-default"),+        ("agent", "propagate"),+        ("server", "custom"),+        ("server", "explicit-default"),+        ("server", "propagate"),+    ],+)+async def test_mcp_tool_retains_factory_failure_policy(+    level: str, policy: str, caplog: pytest.LogCaptureFixture+) -> None:+    from agents.exceptions import AgentsException++    from .mcp.helpers import FakeMCPServer++    class FailingServer(FakeMCPServer):+        async def call_tool(self, tool_name: str, arguments: Any, **kwargs: Any) -> Any:+            raise ValueError(SECRET)++    handler = (+        (lambda ctx, error: "per-tool")+        if policy == "custom"+        else default_tool_error_function+        if policy == "explicit-default"+        else None+    )+    server_options = {"failure_error_function": handler} if level == "server" else {}+    server = FailingServer(**server_options)+    server.add_tool("lookup", {})+    mcp_config: Any = {}+    if level == "server":+        # Server policy must win over an agent-wide MCP policy as well as the run formatter.+        mcp_config["failure_error_function"] = lambda ctx, error: "agent policy"+    elif policy != "omitted":+        mcp_config["failure_error_function"] = handler+    model = make_model("lookup")+    agent = Agent(name="test", model=model, mcp_servers=[server], mcp_config=mcp_config)+    called: list[str] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        called.append(args.tool_name)+        return APPROVED++    config = RunConfig(tool_error_formatter=formatter, trace_include_sensitive_data=True)+    if policy == "propagate":+        with pytest.raises(AgentsException, match=SECRET):+            await Runner.run(agent, "start", run_config=config)+    else:+        with caplog.at_level(logging.DEBUG, logger="openai.agents"):+            result = await Runner.run(agent, "start", run_config=config)+        assert (+            model_output(model)+            == {+                "omitted": GENERIC,+                "custom": "per-tool",+                "explicit-default": GENERIC,+            }[policy]+        )+        if policy == "omitted":+            assert_redacted(caplog, result, model)+    assert not called+++@pytest.mark.asyncio+async def test_run_formatter_uses_qualified_tool_identity() -> None:+    from agents import tool_namespace++    @tool+    async def lookup() -> str:+        raise ValueError(SECRET)++    tools = [+        *tool_namespace(name="crm", description="CRM", tools=[lookup]),+        *tool_namespace(name="billing", description="Billing", tools=[lookup]),+    ]+    model = ScriptedModel(+        [+            [+                get_function_tool_call("lookup", "{}", namespace="crm", call_id="crm-call"),+                get_function_tool_call("lookup", "{}", namespace="billing", call_id="billing-call"),+            ],+            [get_text_message("done")],+        ]+    )++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        return APPROVED if args.tool_name == "crm.lookup" else None++    await Runner.run(+        Agent(name="test", model=model, tools=tools),+        "start",+        run_config=RunConfig(tool_error_formatter=formatter),+    )+    items = model.calls[-1].input+    assert isinstance(items, list)+    assert {+        item["call_id"]: item["output"]+        for item in items+        if item.get("type") == "function_call_output"+    } == {+        "crm-call": APPROVED,+        "billing-call": GENERIC,+    }+++@pytest.mark.asyncio+async def test_run_formatter_collapses_deferred_top_level_namespace() -> None:+    @tool(defer_loading=True)+    async def lookup() -> str:+        raise ValueError(SECRET)++    model = ScriptedModel(+        [+            [get_function_tool_call("lookup", "{}", namespace="lookup")],+            [get_text_message("done")],+        ]+    )++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str | None:+        return APPROVED if args.tool_name == "lookup" else None++    await Runner.run(+        Agent(name="test", model=model, tools=[lookup]),+        "start",+        run_config=RunConfig(tool_error_formatter=formatter),+    )+    assert model_output(model) == APPROVED+++@pytest.mark.asyncio+async def test_mcp_cancellation_keeps_existing_failure_policy() -> None:+    from .mcp.helpers import FakeMCPServer++    class CancelledServer(FakeMCPServer):+        async def call_tool(self, tool_name: str, arguments: Any, **kwargs: Any) -> Any:+            raise asyncio.CancelledError(SECRET)++    server = CancelledServer()+    server.add_tool("lookup", {})+    called: list[ToolErrorFormatterArgs[Any]] = []++    def formatter(args: ToolErrorFormatterArgs[Any]) -> str:+        called.append(args)+        return APPROVED++    model = make_model("lookup")+    result = await Runner.run(+        Agent(name="test", model=model, mcp_servers=[server]),+        "start",+        run_config=RunConfig(tool_error_formatter=formatter),+    )+    assert result.final_output == "done"+    assert not called+    assert model_output(model) == GENERIC