Backtestslitellmrun-20261019-033420#26113

Issue #26113 Merged fix PR # Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: remove_custom_field_from_tools() strips defer_loading — breaks Tool Search on Bedrock, wastes ~20k tokens per request

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Patch B deletes `remove_custom_field_from_tools` from `litellm/llms/bedrock/common_utils.py` but leaves the imports and call sites intact in both `chat/invoke_transformations/anthropic_claude3_transformation.py` and `messages/invoke_transformations/anthropic_claude3_transformation.py`. Those modules now raise ImportError on load, which is exactly what the independent tests show (FAIL-EXC with 'cannot import name remove_custom_field_from_tools'), and the HTTP-level test fails because the request never reaches the mocked transport. So Patch B does not fix the issue; it breaks the Bedrock Anthropic invoke paths entirely. Patch A removes the helper, removes both imports and both call sites, and replaces the old test (which asserted the stripping behaviour) with tests on both transformation paths asserting that `custom: {defer_loading: true}` is preserved while tools without it are untouched. All three independent tests pass under A and all fail on base, confirming it resolves the reported behaviour at the root cause. Scope is tight and matches the issue's suggested fix; craft is fine (dead imports and stale comments cleaned up).

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

15m 10s

Wall-clock time in the container

Full test suite

No new failures

293 passed · 0 skipped · 293 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

129vs 21 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

PR #22861 (merged 2026-03-05, fixing #22847) added remove_custom_field_from_tools() to strip the custom: {defer_loading: true} field from tool definitions before forwarding to Bedrock. At the time, Bedrock rejected this field with Extra inputs are not permitted.

Bedrock has since updated and now accepts custom.defer_loading. When using Claude Code directly against Bedrock (without LiteLLM), Tool Search works correctly — deferred MCP tools are excluded from the system prompt prefix and only ~1.2% of context is consumed by MCP tool definitions.

However, when routing through LiteLLM → Bedrock, remove_custom_field_from_tools() unconditionally strips the custom field. Bedrock then sees all tools as non-deferred and loads every tool definition into the context. This causes ~20k+ extra input tokens per request for a typical MCP setup (22 tools across 5 MCP servers).

Impact

PathMCP tools token costBehavior
Claude Code → Bedrock (direct)~12k tokens (1.2% of context)defer_loading preserved, Tool Search works
Claude Code → LiteLLM → Bedrock~30k+ tokensdefer_loading stripped, all tools fully loaded

This is not just a logging/display issue — it's real additional token consumption and cost on every request.

Root cause

remove_custom_field_from_tools() in litellm/llms/bedrock/common_utils.py unconditionally removes the custom field:

  • litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
  • litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py

Suggested fix

Remove remove_custom_field_from_tools() entirely, or make it conditional/configurable. Bedrock now accepts this field natively.

Steps to Reproduce

  1. Configure Claude Code to use LiteLLM proxy pointing to Bedrock
  2. Add multiple MCP servers (e.g., exa, aws-knowledge, aws-pricing, context7, calculator — 22 tools total)
  3. Start a conversation with a simple prompt like "hi"
  4. Compare token usage in LiteLLM logs vs direct Bedrock connection
  5. LiteLLM path shows ~30k+ tokens for messages; direct Bedrock shows ~2.5k

Environment

  • LiteLLM: v1.82.3-stable (Production)
  • Claude Code: latest
  • Bedrock region: ap-northeast-1
  • Model: Claude Opus 4.6 / Sonnet 4.6

Relevant log output

In LiteLLM request logs, the forwarded request to Bedrock contains tool definitions without custom.defer_loading, while the original request from Claude Code includes it.

Are you a LiteLLM Enterprise user?

No

Twitter / LinkedIn details

_No response_

The two patches

Our patch+62 −67
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.pyindex cff415d49e..b7cc703a23 100644@@ -17,7 +17,6 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation from litellm.llms.bedrock.common_utils import (     get_anthropic_beta_from_headers,     normalize_tool_input_schema_types_for_bedrock_invoke,-    remove_custom_field_from_tools, ) from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER from litellm.types.llms.openai import AllMessageValues@@ -173,8 +172,6 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):         if "anthropic_version" not in anthropic_request:             anthropic_request["anthropic_version"] = self.anthropic_version -        # Remove `custom` field from tools (Bedrock doesn't support it)-        remove_custom_field_from_tools(anthropic_request)         normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_request)         return anthropic_request litellm/llms/bedrock/common_utils.pyindex 52697d752b..3e462cf73a 100644@@ -49,27 +49,6 @@ def get_cached_model_info():     return _get_model_info  -def remove_custom_field_from_tools(request_body: dict) -> None:-    """-    Remove ``custom`` field from each tool in the request body.--    Claude Code (v2.1.69+) sends ``custom: {defer_loading: true}`` on tool-    definitions, which Anthropic's API accepts but Bedrock rejects with-    ``"Extra inputs are not permitted"``.--    Args:-        request_body: The request dictionary to modify in-place.--    Ref: https://github.com/BerriAI/litellm/issues/22847-    """-    tools = request_body.get("tools")-    if not tools or not isinstance(tools, list):-        return-    for tool in tools:-        if isinstance(tool, dict):-            tool.pop("custom", None)-- def normalize_json_schema_custom_types_to_object(schema: dict) -> None:     """     In-place: replace JSON Schema ``type: \"custom\"`` with ``\"object\"`` (iterative walk).litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.pyindex 31e0e76fd9..709eefabe2 100644@@ -31,7 +31,6 @@ from litellm.llms.bedrock.common_utils import (     get_anthropic_beta_from_headers,     is_claude_4_5_on_bedrock,     normalize_tool_input_schema_types_for_bedrock_invoke,-    remove_custom_field_from_tools, ) from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER from litellm.types.llms.openai import AllMessageValues@@ -504,11 +503,6 @@ class AmazonAnthropicClaudeMessagesConfig(         # Fixes: https://github.com/BerriAI/litellm/issues/22797         anthropic_messages_request.pop("output_config", None) -        # 5a. Remove `custom` field from tools (Bedrock doesn't support it)-        # Claude Code sends `custom: {defer_loading: true}` on tool definitions,-        # which causes Bedrock to reject the request with "Extra inputs are not permitted"-        # Ref: https://github.com/BerriAI/litellm/issues/22847-        remove_custom_field_from_tools(anthropic_messages_request)         normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_messages_request)         ensure_bedrock_anthropic_messages_tool_names(anthropic_messages_request) tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.pyindex 80d6d26e7e..aafe544ed3 100644@@ -475,3 +475,41 @@ def test_output_format_removed_from_bedrock_invoke_request():     assert (         "output_format" not in result     ), f"output_format should be removed for Bedrock Invoke, got keys: {result.keys()}"+++def test_bedrock_chat_invoke_preserves_custom_defer_loading_on_tools():+    """+    The chat/invoke path must forward ``custom: {defer_loading: true}`` on+    Anthropic-format tool definitions to Bedrock so Tool Search keeps+    deferred tools out of the context.+    """+    config = AmazonAnthropicClaudeConfig()++    optional_params = {+        "max_tokens": 100,+        "tools": [+            {+                "name": "mcp__exa__search",+                "description": "Search the web",+                "input_schema": {"type": "object", "properties": {}},+                "custom": {"defer_loading": True},+            },+            {+                "name": "Read",+                "description": "Read a file",+                "input_schema": {"type": "object", "properties": {}},+            },+        ],+    }++    result = config.transform_request(+        model="anthropic.claude-sonnet-4-5-20250929-v1:0",+        messages=[{"role": "user", "content": "hi"}],+        optional_params=optional_params,+        litellm_params={},+        headers={},+    )++    tools_by_name = {tool["name"]: tool for tool in result["tools"]}+    assert tools_by_name["mcp__exa__search"]["custom"] == {"defer_loading": True}+    assert "custom" not in tools_by_name["Read"]tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.pyindex d3a9c94ea5..f3c98d73f8 100644@@ -16,7 +16,6 @@ from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging from litellm.llms.bedrock.common_utils import (     ensure_bedrock_anthropic_messages_tool_names,     normalize_tool_input_schema_types_for_bedrock_invoke,-    remove_custom_field_from_tools, ) from litellm.constants import BEDROCK_MIN_THINKING_BUDGET_TOKENS from litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation import (@@ -248,56 +247,44 @@ def test_remove_ttl_from_cache_control():     assert request5 == {}  -def test_remove_custom_field_from_tools():+def test_bedrock_invoke_messages_preserves_custom_defer_loading_on_tools():     """-    Ensure the `custom` field is stripped from every tool definition.--    Claude Code v2.1.69+ sends `custom: {defer_loading: true}` on tool-    objects.  Bedrock does not accept this extra field and returns-    "Extra inputs are not permitted".--    Ref: https://github.com/BerriAI/litellm/issues/22847+    Claude Code sends ``custom: {defer_loading: true}`` on tool definitions to+    enable Tool Search. Bedrock accepts this field, so it must be forwarded+    unchanged; stripping it makes Bedrock load every deferred tool definition+    into the context.     """+    from litellm.types.router import GenericLiteLLMParams -    # Case 1: tool with `custom` field should have it removed-    request = {+    cfg = AmazonAnthropicClaudeMessagesConfig()+    optional_params = {+        "max_tokens": 256,         "tools": [             {-                "name": "Read",-                "description": "Read a file",+                "name": "mcp__exa__search",+                "description": "Search the web",                 "input_schema": {"type": "object", "properties": {}},                 "custom": {"defer_loading": True},             },             {-                "name": "Write",-                "description": "Write a file",+                "name": "Read",+                "description": "Read a file",                 "input_schema": {"type": "object", "properties": {}},             },-        ]+        ],+        "stream": False,     } -    remove_custom_field_from_tools(request)--    for tool in request["tools"]:-        assert "custom" not in tool, f"Tool {tool['name']} still has 'custom' field"-    # Other fields should be preserved-    assert request["tools"][0]["name"] == "Read"-    assert request["tools"][1]["name"] == "Write"--    # Case 2: request without tools key (should not raise error)-    request2 = {"messages": [{"role": "user", "content": "hi"}]}-    remove_custom_field_from_tools(request2)-    assert "tools" not in request2--    # Case 3: empty tools list (should not raise error)-    request3 = {"tools": []}-    remove_custom_field_from_tools(request3)-    assert request3["tools"] == []+    result = cfg.transform_anthropic_messages_request(+        model="anthropic.claude-sonnet-4-5-20250929-v1:0",+        messages=[{"role": "user", "content": "hi"}],+        anthropic_messages_optional_request_params=optional_params,+        litellm_params=GenericLiteLLMParams(),+        headers={},+    ) -    # Case 4: tools with None value (should not raise error)-    request4 = {"tools": None}-    remove_custom_field_from_tools(request4)-    assert request4["tools"] is None+    assert result["tools"][0]["custom"] == {"defer_loading": True}+    assert "custom" not in result["tools"][1]   def test_normalize_tool_input_schema_types_for_bedrock_invoke(): 
Merged fix · PR #+0 −21
litellm/llms/bedrock/common_utils.pyindex 52697d752be..3e462cf73aa 100644@@ -49,27 +49,6 @@ def get_cached_model_info():     return _get_model_info  -def remove_custom_field_from_tools(request_body: dict) -> None:-    """-    Remove ``custom`` field from each tool in the request body.--    Claude Code (v2.1.69+) sends ``custom: {defer_loading: true}`` on tool-    definitions, which Anthropic's API accepts but Bedrock rejects with-    ``"Extra inputs are not permitted"``.--    Args:-        request_body: The request dictionary to modify in-place.--    Ref: https://github.com/BerriAI/litellm/issues/22847-    """-    tools = request_body.get("tools")-    if not tools or not isinstance(tools, list):-        return-    for tool in tools:-        if isinstance(tool, dict):-            tool.pop("custom", None)-- def normalize_json_schema_custom_types_to_object(schema: dict) -> None:     """     In-place: replace JSON Schema ``type: \"custom\"`` with ``\"object\"`` (iterative walk).