Backtestspydantic-airun-20260918-215936#7791
BedrockConverseModel ignores anthropic_disallows_sampling_settings, so temperature/top_p/top_k hard-fail on Claude Opus 5 / Sonnet 5
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the root cause identically and soundly: promote the sampling-param tuple into profiles/anthropic.py (avoiding the anthropic-SDK import cycle in bedrock.py), then filter temperature/top_p/top_k from the merged settings inside BedrockConverseModel.prepare_request when profile.anthropic_disallows_sampling_settings is set. Both pop top_k, which correctly neutralizes the additionalModelRequestFields injection leg (bedrock.py:956 only adds top_k when it's present in settings) as well as the inferenceConfig leg, and both filter a copy so the caller's dict is untouched and both collapse an emptied dict to None. So correctness (criterion 1) is a wash. The decision turns on tests (criterion 2), where B is clearly stronger: A ships a single unit test asserting on prepare_request's returned dict, which never proves top_k actually stops reaching additionalModelRequestFields on the wire. B asserts on the recorded request body (inferenceConfig=={'maxTokens':16}, additionalModelRequestFields absent), adds a gating test proving a non-flagged model (Haiku 4.5) still forwards temperature+top_k — directly closing the coverage gap the issue names — plus a count_tokens test (that path also routes through prepare_request) and a no-op/no-warning case, and a raw-client cassette documenting the actual 400 so the fix's premise can't silently drift. Craft also favors B: its bedrock _drop_unsupported_sampling_settings mirrors AnthropicModel's in-place signature and its docstring correctly explains both the import-cycle placement and the deliberate decision to leave the bedrock_additional_model_requests_fields escape hatch alone; A leaves a slightly awkward `_ANTHROPIC_SAMPLING_PARAMS = ANTHROPIC_SAMPLING_PARAMS` alias lingering rather than migrating call sites. A wins only on scope (smaller diff, less churn in anthropic.py), but the extra scope in B is almost entirely justified test infrastructure (cassettes) and a clean constant rename. Since tests and craft outrank scope, B is the better fix.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
18m 48s4 turns
Wall-clock time in the container
Full test suite
No new failures
12,141 passed · 582 skipped · 12,845 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
70vs 312 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I confirm that I'm using the latest version of Pydantic AI (hit in production on
2.31.1; enforcement gap re-verified in the latest releasev2.35.0and on currentmain) - I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues/ before opening this issue
Description
AnthropicModel honors anthropic_disallows_sampling_settings (warn + drop temperature/top_p/top_k, models/anthropic.py:857), but BedrockConverseModel never reads the flag, so every request with a sampling setting hard-fails with a 400 on the Claude 5 generation over Bedrock:
pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: eu.anthropic.claude-opus-5,
body: {'Error': {'Message': 'The model returned the following errors: `temperature` is deprecated for this model.', 'Code': 'ValidationException'}, ...}
The same Agent(..., model_settings={'temperature': 0.5}) on anthropic:claude-opus-5 works (settings are dropped with a UserWarning). A global temperature in shared model_settings therefore breaks an app the moment a model is switched to Opus 5 / Sonnet 5 on Bedrock, which is how we hit it in production.
Live probe matrix (raw bedrock-runtime converse, eu-central-1, 2026-08-26):
| model | temperature | top_p | top_k (additionalModelRequestFields) |
|---|---|---|---|
eu.anthropic.claude-opus-5 | 400 "deprecated for this model" | 400 | 400 |
eu.anthropic.claude-sonnet-5 | 400 | 400 | not probed |
eu.anthropic.claude-haiku-4-5-... | ok | ok | ok |
Root cause (current main):
- The profile is resolved correctly:
BedrockProvider.model_profile('eu.anthropic.claude-opus-5')carriesanthropic_disallows_sampling_settings=True(providers/bedrock.py:248-289merges the downstream Anthropic profile). It's one missed read, not a missing profile. models/bedrock.py:1062-1076_map_inference_configmapstemperature/top_punconditionally, and as a@staticmethodit can't even seeself.profile.models/bedrock.py:928-930injectstop_kintoadditionalModelRequestFieldsfor every Anthropic model (bedrock_top_k_variant == 'anthropic'), same failure.- Repo-wide,
models/anthropic.py:857is the flag's only enforcement site;models/bedrock.py/providers/bedrock.pynever mention it. #6723 verified the rejection against the direct API and deliberately scoped Bedrock out ("No Bedrock/Heroku model ids"); #7283 already names the pattern ("the Anthropic fix wasn't widened to Bedrock" pending sibling reproduction), which the probes above now provide.
This also contradicts the docs: docs/models/anthropic.md promises "Pydantic AI drops those keys automatically" for Opus 4.7/4.8/5 with no Bedrock exclusion (unlike e.g. task budgets, which are explicitly scoped to "not Bedrock, Vertex, or Microsoft Foundry"), and the ModelSettings reference lists Bedrock under temperature/top_p.
Proposed fix (bounded): mirror AnthropicModel in BedrockConverseModel.prepare_request (models/bedrock.py:666-704): when profile.get('anthropic_disallows_sampling_settings'), pop temperature/top_p/top_k from the prepared settings with the same UserWarning ("These settings will be ignored."). Popping top_k there also covers the additionalModelRequestFields leg. Promote _ANTHROPIC_SAMPLING_PARAMS (models/anthropic.py:316) to profiles/anthropic.py, whose docstring already declares it shared with "any provider that translates to the Anthropic wire shape (e.g. Bedrock Converse)", and import it next to ANTHROPIC_THINKING_BUDGET_MAP in models/bedrock.py:79. Add an Opus-5 Bedrock case to tests/models/test_model_settings_support.py (the existing Bedrock case at :373-394 uses Sonnet 4.5, whose flag is False, so this combination has no coverage).
Open design question for maintainers: is prepare_request the right site, or would you rather make _map_inference_config profile-aware / add a bedrock_* mirror flag in bedrock_anthropic_model_profile like the existing bedrock_supports_adaptive_thinking mirror?
Noted while auditing, deliberately not claimed here (separate issues if you want them, per AGENTS.md sibling rule): resolve_anthropic_effort(..., supports_xhigh=False) is hardcoded at models/bedrock.py:955 (silent xhigh→max downgrade), and bedrock_supports_tool_choice is hardcoded True for all Anthropic ids (providers/bedrock.py:268), which puts forced toolChoice on the wire for claude-fable-5/claude-mythos-5 Bedrock ids.
Workaround: use AnthropicModel with AsyncAnthropicBedrock (that route resolves the same profile and goes through the working prepare_request), or unset temperature/top_p for these models.
Happy to open a PR once a maintainer agrees on the approach and assigns this.
(Root-cause analysis and drafting done with Claude Code; the Bedrock probes above were run against a live account and reviewed by me.)
Example Code
from pydantic_ai import Agent
agent = Agent(
'bedrock:eu.anthropic.claude-opus-5',
model_settings={'temperature': 0.5}, # or top_p / top_k
)
agent.run_sync('Say OK')
# ModelHTTPError 400: '`temperature` is deprecated for this model.'
# Same settings on 'anthropic:claude-opus-5' warn + drop instead.
Python, Pydantic AI & LLM client version
- Python: 3.12
- Pydantic AI: 2.31.1 in production; re-verified on v2.35.0 (models/bedrock.py:1069, no reference to the flag) and current main (:1062-1076)
- LLM provider SDK: boto3 1.40.61
- AWS: eu-central-1; model ids eu.anthropic.claude-opus-5, eu.anthropic.claude-sonnet-5The two patches
pydantic_ai_slim/pydantic_ai/models/anthropic.pyindex e70b2de2..48ef9cf8 100644@@ -71,6 +71,7 @@ from ..native_tools._tool_search import ( ) from ..profiles import DEFAULT_THINKING_TAGS, ModelProfile, ModelProfileSpec, merge_profile from ..profiles.anthropic import (+ ANTHROPIC_SAMPLING_PARAMS, ANTHROPIC_THINKING_BUDGET_MAP, AnthropicCodeExecutionToolVersion, AnthropicEffort,@@ -327,7 +328,7 @@ _ADVISOR_UNSUPPORTED_CLIENTS = (AsyncAnthropicBedrock, AsyncAnthropicVertex, Asy # that ignores the entry on *every* transport — which measured the model, not the transport. _INLINE_SYSTEM_PROMPT_UNSUPPORTED_CLIENTS = (AsyncAnthropicFoundry,) -_ANTHROPIC_SAMPLING_PARAMS = ('temperature', 'top_p', 'top_k')+_ANTHROPIC_SAMPLING_PARAMS = ANTHROPIC_SAMPLING_PARAMS _ANTHROPIC_TASK_BUDGETS_BETA = 'task-budgets-2026-03-13' _ANTHROPIC_THINKING_BINDING_BETA = 'thinking-binding-controls-2026-08-01' _ANTHROPIC_DROP_STALE_THINKING_BLOCKS: BetaThinkingBlockBindingParam = {'prefix_mismatch_behavior': 'drop_block'}pydantic_ai_slim/pydantic_ai/models/bedrock.pyindex 83631543..90c19570 100644@@ -79,7 +79,11 @@ from pydantic_ai.models import ( from pydantic_ai.models._tool_choice import ResolvedToolChoice, resolve_tool_choice from pydantic_ai.native_tools import AbstractNativeTool, CodeExecutionTool from pydantic_ai.profiles import DEFAULT_THINKING_TAGS-from pydantic_ai.profiles.anthropic import ANTHROPIC_THINKING_BUDGET_MAP, resolve_anthropic_effort+from pydantic_ai.profiles.anthropic import (+ ANTHROPIC_SAMPLING_PARAMS,+ ANTHROPIC_THINKING_BUDGET_MAP,+ resolve_anthropic_effort,+) from pydantic_ai.profiles.openai import OPENAI_REASONING_EFFORT_MAP from pydantic_ai.providers import Provider, infer_provider from pydantic_ai.providers.bedrock import BedrockModelProfile, remove_bedrock_geo_prefix@@ -706,7 +710,25 @@ class BedrockConverseModel(Model[BaseClient]): model_request_parameters, output_object=replace(model_request_parameters.output_object, strict=True) ) # Pass unmerged model_settings; base class does its own merge- return super().prepare_request(model_settings, model_request_parameters)+ prepared_settings, model_request_parameters = super().prepare_request(+ model_settings, model_request_parameters+ )+ if self.profile.get('anthropic_disallows_sampling_settings', False) and prepared_settings:+ prepared_settings = self._drop_unsupported_sampling_settings(prepared_settings)+ return prepared_settings, model_request_parameters++ def _drop_unsupported_sampling_settings(self, model_settings: ModelSettings) -> ModelSettings | None:+ filtered: ModelSettings = {**model_settings}+ dropped = [setting for setting in ANTHROPIC_SAMPLING_PARAMS if setting in filtered]+ for setting in dropped:+ filtered.pop(setting, None)+ if dropped:+ warnings.warn(+ f'Sampling parameters {dropped} are not supported by {self.model_name!r}. These settings will be ignored.',+ UserWarning,+ stacklevel=2,+ )+ return filtered or None @property def _botocore_supports_strict_tool_param(self) -> bool:pydantic_ai_slim/pydantic_ai/profiles/anthropic.pyindex ba211cd6..555fc090 100644@@ -14,6 +14,13 @@ from ..native_tools._tool_search import ToolSearchTool from ..settings import ThinkingEffort, ThinkingLevel from . import ModelProfile +ANTHROPIC_SAMPLING_PARAMS = ('temperature', 'top_p', 'top_k')+"""Sampling settings rejected by models with `anthropic_disallows_sampling_settings`.++Shared between the direct Anthropic path and any provider that translates to the+Anthropic wire shape (e.g. Bedrock Converse for Anthropic models).+"""+ _ANTHROPIC_BASE_BUILTINS = frozenset({WebSearchTool, CodeExecutionTool, WebFetchTool, MemoryTool, MCPServerTool}) """Native tool types Anthropic generally supports across the model line. Mirrors `AnthropicModel.supported_native_tools()` minus `ToolSearchTool`, which is gatedtests/models/test_bedrock.pyindex fa56d2b7..9deb2ad4 100644@@ -3475,6 +3475,40 @@ async def test_bedrock_capability_tools_stay_off_the_wire(bedrock_provider: Bedr ) +async def test_anthropic_disallows_sampling_settings_dropped(+ allow_model_requests: None, bedrock_provider: BedrockProvider+):+ """Claude Opus 5 / Sonnet 5 over Bedrock inherit `anthropic_disallows_sampling_settings`.++ `prepare_request` must drop `temperature`/`top_p`/`top_k` (with a `UserWarning`) instead of sending+ them on the wire, mirroring `AnthropicModel`, and must not mutate the caller's `model_settings`.+ """+ model = BedrockConverseModel('eu.anthropic.claude-opus-5', provider=bedrock_provider)+ assert model.profile.get('anthropic_disallows_sampling_settings') is True++ model_settings: dict[str, Any] = {'temperature': 0.5, 'top_p': 0.9, 'top_k': 10, 'max_tokens': 100}++ with pytest.warns(UserWarning, match='Sampling parameters') as recorded:+ prepared_settings, _ = model.prepare_request(+ cast(Any, model_settings), ModelRequestParameters()+ )++ sampling_warnings = [str(w.message) for w in recorded if 'Sampling parameters' in str(w.message)]+ assert sampling_warnings == [+ "Sampling parameters ['temperature', 'top_p', 'top_k'] are not supported by "+ "'eu.anthropic.claude-opus-5'. These settings will be ignored."+ ]++ assert prepared_settings is not None+ assert 'temperature' not in prepared_settings+ assert 'top_p' not in prepared_settings+ assert 'top_k' not in prepared_settings+ assert prepared_settings['max_tokens'] == 100++ # Filtering happens on a copy: the caller's original dict keeps all four keys.+ assert model_settings == {'temperature': 0.5, 'top_p': 0.9, 'top_k': 10, 'max_tokens': 100}++ async def test_bedrock_delta_renders_announcement_and_plain_tool_spec( allow_model_requests: None, bedrock_provider: BedrockProvider, mocker: MockerFixture ) -> None:
pydantic_ai_slim/pydantic_ai/models/anthropic.pyindex e70b2de2ac..eee0324c17 100644@@ -71,6 +71,7 @@ from ..native_tools._tool_search import ( ) from ..profiles import DEFAULT_THINKING_TAGS, ModelProfile, ModelProfileSpec, merge_profile from ..profiles.anthropic import (+ ANTHROPIC_SAMPLING_PARAMS, ANTHROPIC_THINKING_BUDGET_MAP, AnthropicCodeExecutionToolVersion, AnthropicEffort,@@ -327,7 +328,6 @@ _ADVISOR_UNSUPPORTED_CLIENTS = (AsyncAnthropicBedrock, AsyncAnthropicVertex, Asy # that ignores the entry on *every* transport — which measured the model, not the transport. _INLINE_SYSTEM_PROMPT_UNSUPPORTED_CLIENTS = (AsyncAnthropicFoundry,) -_ANTHROPIC_SAMPLING_PARAMS = ('temperature', 'top_p', 'top_k') _ANTHROPIC_TASK_BUDGETS_BETA = 'task-budgets-2026-03-13' _ANTHROPIC_THINKING_BINDING_BETA = 'thinking-binding-controls-2026-08-01' _ANTHROPIC_DROP_STALE_THINKING_BLOCKS: BetaThinkingBlockBindingParam = {'prefix_mismatch_behavior': 'drop_block'}@@ -596,7 +596,7 @@ def _build_extra_body( the SDK gave it while the parameters were still named arguments. """ fields: dict[str, Any] = {- setting: value for setting in _ANTHROPIC_SAMPLING_PARAMS if (value := model_settings.get(setting)) is not None+ setting: value for setting in ANTHROPIC_SAMPLING_PARAMS if (value := model_settings.get(setting)) is not None } if thinking_override is not None: fields['thinking'] = thinking_override@@ -1086,19 +1086,19 @@ class AnthropicModel(Model[AsyncAnthropicClient]): return prepared_settings, model_request_parameters def _drop_unsupported_sampling_settings(self, model_settings: ModelSettings) -> None:- dropped = {setting for setting in _ANTHROPIC_SAMPLING_PARAMS if setting in model_settings}+ dropped = {setting for setting in ANTHROPIC_SAMPLING_PARAMS if setting in model_settings} extra_body = model_settings.get('extra_body') if is_str_dict(extra_body):- dropped |= {setting for setting in _ANTHROPIC_SAMPLING_PARAMS if setting in extra_body}+ dropped |= {setting for setting in ANTHROPIC_SAMPLING_PARAMS if setting in extra_body} model_settings['extra_body'] = {- key: value for key, value in extra_body.items() if key not in _ANTHROPIC_SAMPLING_PARAMS+ key: value for key, value in extra_body.items() if key not in ANTHROPIC_SAMPLING_PARAMS } - for setting in _ANTHROPIC_SAMPLING_PARAMS:+ for setting in ANTHROPIC_SAMPLING_PARAMS: model_settings.pop(setting, None) if dropped:- ordered = [setting for setting in _ANTHROPIC_SAMPLING_PARAMS if setting in dropped]+ ordered = [setting for setting in ANTHROPIC_SAMPLING_PARAMS if setting in dropped] warnings.warn( f'Sampling parameters {ordered} are not supported by {self.model_name!r}. These settings will be ignored.', UserWarning,pydantic_ai_slim/pydantic_ai/models/bedrock.pyindex 83631543f2..86e94b9987 100644@@ -79,7 +79,11 @@ from pydantic_ai.models import ( from pydantic_ai.models._tool_choice import ResolvedToolChoice, resolve_tool_choice from pydantic_ai.native_tools import AbstractNativeTool, CodeExecutionTool from pydantic_ai.profiles import DEFAULT_THINKING_TAGS-from pydantic_ai.profiles.anthropic import ANTHROPIC_THINKING_BUDGET_MAP, resolve_anthropic_effort+from pydantic_ai.profiles.anthropic import (+ ANTHROPIC_SAMPLING_PARAMS,+ ANTHROPIC_THINKING_BUDGET_MAP,+ resolve_anthropic_effort,+) from pydantic_ai.profiles.openai import OPENAI_REASONING_EFFORT_MAP from pydantic_ai.providers import Provider, infer_provider from pydantic_ai.providers.bedrock import BedrockModelProfile, remove_bedrock_geo_prefix@@ -706,7 +710,32 @@ class BedrockConverseModel(Model[BaseClient]): model_request_parameters, output_object=replace(model_request_parameters.output_object, strict=True) ) # Pass unmerged model_settings; base class does its own merge- return super().prepare_request(model_settings, model_request_parameters)+ prepared_settings, model_request_parameters = super().prepare_request(model_settings, model_request_parameters)+ if self.profile.get('anthropic_disallows_sampling_settings', False) and prepared_settings:+ filtered: ModelSettings = {**prepared_settings}+ self._drop_unsupported_sampling_settings(filtered)+ prepared_settings = filtered or None+ return prepared_settings, model_request_parameters++ def _drop_unsupported_sampling_settings(self, model_settings: ModelSettings) -> None:+ """Drop the sampling settings a flagged model rejects, warning like `AnthropicModel` does.++ `temperature` and `top_p` would reach `inferenceConfig` and unified `top_k` would reach+ `additionalModelRequestFields`; all three are rejected with a 400 by the models that set+ `anthropic_disallows_sampling_settings`. A user's own+ `bedrock_additional_model_requests_fields` is deliberately left alone: it is the raw+ escape hatch, so a value placed there is addressed to Bedrock directly.+ """+ dropped = [setting for setting in ANTHROPIC_SAMPLING_PARAMS if setting in model_settings]+ for setting in dropped:+ model_settings.pop(setting, None)++ if dropped:+ warnings.warn(+ f'Sampling parameters {dropped} are not supported by {self.model_name!r}. These settings will be ignored.',+ UserWarning,+ stacklevel=2,+ ) @property def _botocore_supports_strict_tool_param(self) -> bool:pydantic_ai_slim/pydantic_ai/profiles/anthropic.pyindex ba211cd62b..48bdd4c635 100644@@ -19,6 +19,15 @@ _ANTHROPIC_BASE_BUILTINS = frozenset({WebSearchTool, CodeExecutionTool, WebFetch `AnthropicModel.supported_native_tools()` minus `ToolSearchTool`, which is gated per-model in the profile below.""" +ANTHROPIC_SAMPLING_PARAMS = ('temperature', 'top_p', 'top_k')+"""The unified sampling settings gated by `anthropic_disallows_sampling_settings`.++Models whose profile sets that flag reject these settings outright with a 400, so every provider+serving them has to drop and warn rather than forward. It lives here rather than beside either+model because `models/bedrock.py` cannot import from `models/anthropic.py` without pulling the+`anthropic` SDK into the `bedrock` extra.+"""+ AnthropicCodeExecutionToolVersion: TypeAlias = Literal['20250825', '20260120'] """Concrete Anthropic code execution tool version to send for `CodeExecutionTool`.""" tests/models/cassettes/test_bedrock/test_bedrock_anthropic_5_api_rejects_sampling_settings.yamlnew file mode 100644index 0000000000..872d80ab47@@ -0,0 +1,32 @@+interactions:+- request:+ body: '{"messages": [{"role": "user", "content": [{"text": "What is 2+2?"}]}], "inferenceConfig": {"maxTokens": 16, "temperature":+ 0.2}}'+ headers:+ amz-sdk-invocation-id:+ - !!binary |+ YTEzNGQzOTgtYmVkZi00NTI3LWFhZjgtNDVlYmM0MDBlZTZk+ amz-sdk-request:+ - !!binary |+ YXR0ZW1wdD0x+ content-length:+ - '129'+ content-type:+ - !!binary |+ YXBwbGljYXRpb24vanNvbg==+ method: POST+ uri: https://bedrock-runtime.eu-central-1.amazonaws.com/model/eu.anthropic.claude-opus-5/converse+ response:+ headers:+ connection:+ - keep-alive+ content-length:+ - '99'+ content-type:+ - application/json+ parsed_body:+ message: 'The model returned the following errors: `temperature` is deprecated for this model.'+ status:+ code: 400+ message: Bad Request+version: 1tests/models/cassettes/test_bedrock/test_bedrock_anthropic_5_drops_sampling_settings.yamlnew file mode 100644index 0000000000..927fe31ab4@@ -0,0 +1,46 @@+interactions:+- request:+ body: '{"messages": [{"role": "user", "content": [{"text": "What is 2+2? Answer with the number only."}]}], "system":+ [], "inferenceConfig": {"maxTokens": 16}}'+ headers:+ amz-sdk-invocation-id:+ - !!binary |+ ZTBiN2JhOWItODZiOC00MWE5LWE3YjItZDFjZmNhNTQ0YjMw+ amz-sdk-request:+ - !!binary |+ YXR0ZW1wdD0x+ content-length:+ - '152'+ content-type:+ - !!binary |+ YXBwbGljYXRpb24vanNvbg==+ method: POST+ uri: https://bedrock-runtime.eu-central-1.amazonaws.com/model/eu.anthropic.claude-opus-5/converse+ response:+ headers:+ connection:+ - keep-alive+ content-length:+ - '280'+ content-type:+ - application/json+ parsed_body:+ metrics:+ latencyMs: 960+ output:+ message:+ content:+ - text: '4'+ role: assistant+ stopReason: end_turn+ usage:+ cacheReadInputTokenCount: 0+ cacheReadInputTokens: 0+ inputTokens: 21+ outputTokens: 3+ serverToolUsage: {}+ totalTokens: 24+ status:+ code: 200+ message: OK+version: 1tests/models/cassettes/test_bedrock/test_bedrock_non_flagged_model_keeps_sampling_settings.yamlnew file mode 100644index 0000000000..4612963fbb@@ -0,0 +1,46 @@+interactions:+- request:+ body: '{"messages": [{"role": "user", "content": [{"text": "What is 2+2? Answer with the number only."}]}], "system":+ [], "inferenceConfig": {"maxTokens": 16, "temperature": 0.2}, "additionalModelRequestFields": {"top_k": 5}}'+ headers:+ amz-sdk-invocation-id:+ - !!binary |+ Y2M1YjIyNTAtNDNkOS00ZDVkLWFkYTItZjRlN2UyOTk0NjE5+ amz-sdk-request:+ - !!binary |+ YXR0ZW1wdD0x+ content-length:+ - '218'+ content-type:+ - !!binary |+ YXBwbGljYXRpb24vanNvbg==+ method: POST+ uri: https://bedrock-runtime.eu-central-1.amazonaws.com/model/eu.anthropic.claude-haiku-4-5-20251001-v1%3A0/converse+ response:+ headers:+ connection:+ - keep-alive+ content-length:+ - '280'+ content-type:+ - application/json+ parsed_body:+ metrics:+ latencyMs: 458+ output:+ message:+ content:+ - text: '4'+ role: assistant+ stopReason: end_turn+ usage:+ cacheReadInputTokenCount: 0+ cacheReadInputTokens: 0+ inputTokens: 20+ outputTokens: 5+ serverToolUsage: {}+ totalTokens: 25+ status:+ code: 200+ message: OK+version: 1tests/models/test_bedrock.pyindex fa56d2b774..83733af429 100644@@ -6858,3 +6858,135 @@ async def test_bedrock_anthropic_message_history_starting_with_response( ), ] )+++def test_bedrock_anthropic_5_api_rejects_sampling_settings(+ allow_model_requests: None, bedrock_provider: BedrockProvider+):+ """Bedrock itself rejects the sampling settings on a Claude 5 model, which is why they are dropped.++ Sent through the raw client rather than the model, so this records the API's own behavior and+ cannot drift with our filtering: cassettes are matched on the URL, not the body, so a recording+ made through the model would keep replaying if the settings ever started riding along again.+ `test_bedrock_anthropic_5_drops_sampling_settings` is what catches that, by asserting on the+ recorded request body.+ """+ model = BedrockConverseModel('eu.anthropic.claude-opus-5', provider=bedrock_provider)++ with pytest.raises(ClientError) as exc_info:+ model.client.converse(+ modelId='eu.anthropic.claude-opus-5',+ messages=[{'role': 'user', 'content': [{'text': 'What is 2+2?'}]}],+ inferenceConfig={'maxTokens': 16, 'temperature': 0.2},+ )++ # Asserted on the status and message rather than `Error.Code`: botocore derives the code from+ # the `x-amzn-errortype` response header, which the cassette's header filtering does not keep,+ # so on replay it falls back to the HTTP status. `response` is a `TypedDict` whose keys are all+ # optional, so it is read as a plain mapping.+ response = cast(dict[str, Any], exc_info.value.response)+ assert response['ResponseMetadata']['HTTPStatusCode'] == 400+ assert response['Error']['Message'] == snapshot(+ 'The model returned the following errors: `temperature` is deprecated for this model.'+ )+++async def test_bedrock_anthropic_5_drops_sampling_settings(+ allow_model_requests: None, bedrock_provider: BedrockProvider, vcr: Cassette+):+ """A Claude 5 model on Bedrock warns and drops the sampling settings instead of failing with a 400.++ Mirrors `AnthropicModel`, which honors the same `anthropic_disallows_sampling_settings` profile+ flag. Asserted on the recorded request body rather than on the call succeeding: `temperature`+ and `top_p` must not reach `inferenceConfig`, and unified `top_k` must not reach+ `additionalModelRequestFields`.+ """+ settings = BedrockModelSettings(max_tokens=16, temperature=0.2, top_p=0.3, top_k=5)+ model = BedrockConverseModel('eu.anthropic.claude-opus-5', provider=bedrock_provider)+ agent = Agent(model, model_settings=settings)++ with pytest.warns(UserWarning, match='Sampling parameters') as recorded:+ result = await agent.run('What is 2+2? Answer with the number only.')++ assert result.output.strip() == snapshot('4')+ sampling_warnings = [str(w.message) for w in recorded if 'Sampling parameters' in str(w.message)]+ assert sampling_warnings == snapshot(+ [+ "Sampling parameters ['temperature', 'top_p', 'top_k'] are not supported by "+ "'eu.anthropic.claude-opus-5'. These settings will be ignored."+ ]+ )++ sent = single_request_body(vcr)+ assert sent['inferenceConfig'] == snapshot({'maxTokens': 16})+ assert 'additionalModelRequestFields' not in sent+ # Filtering happens on a copy, so the caller's own settings dict is left intact.+ assert settings == snapshot({'max_tokens': 16, 'temperature': 0.2, 'top_p': 0.3, 'top_k': 5})+++async def test_bedrock_non_flagged_model_keeps_sampling_settings(+ allow_model_requests: None, bedrock_provider: BedrockProvider, vcr: Cassette+):+ """The drop is gated on the profile flag: a model without it still receives the settings.++ Without this, a filter that over-reached would look identical to a working one. `top_p` is left+ out because the model rejects it alongside `temperature` with "`temperature` and `top_p` cannot+ both be specified for this model"; all three settings travel the same path, so one of the pair is+ enough to pin it (the same reason `test_anthropic_sampling_settings_reach_the_wire` omits it).+ """+ settings = BedrockModelSettings(max_tokens=16, temperature=0.2, top_k=5)+ model = BedrockConverseModel('eu.anthropic.claude-haiku-4-5-20251001-v1:0', provider=bedrock_provider)+ agent = Agent(model, model_settings=settings)++ await agent.run('What is 2+2? Answer with the number only.')++ sent = single_request_body(vcr)+ assert sent['inferenceConfig'] == snapshot({'maxTokens': 16, 'temperature': 0.2})+ assert sent['additionalModelRequestFields'] == snapshot({'top_k': 5})+++class _CountTokensCapturingClient:+ """Records the `count_tokens` params instead of calling Bedrock."""++ def __init__(self):+ self.calls: list[dict[str, Any]] = []+ self.meta = SimpleNamespace(endpoint_url='https://bedrock.stub', events=HierarchicalEmitter())++ def count_tokens(self, **kwargs: Any) -> dict[str, int]:+ self.calls.append(kwargs)+ return {'inputTokens': 3}+++async def test_bedrock_anthropic_5_count_tokens_drops_sampling_settings(+ allow_model_requests: None, bedrock_provider: BedrockProvider+):+ """`count_tokens` drops them too, because it routes through the same `prepare_request`.++ Captured from a stub client rather than recorded: no model that sets the flag supports Bedrock's+ CountTokens operation today (it answers "The provided model doesn't support counting tokens."),+ so there is no live exchange to record. The profile still comes from the real provider.+ """+ client = _CountTokensCapturingClient()+ settings = BedrockModelSettings(max_tokens=16, temperature=0.2, top_p=0.3, top_k=5)+ model = BedrockConverseModel('eu.anthropic.claude-opus-5', provider=bedrock_provider)+ model.client = cast(Any, client)++ with pytest.warns(UserWarning, match='Sampling parameters'):+ result = await model.count_tokens(+ [ModelRequest.user_text_prompt('Hello, world!')], settings, ModelRequestParameters()+ )++ assert result.input_tokens == 3+ assert 'additionalModelRequestFields' not in client.calls[0]['input']['converse']+++def test_bedrock_anthropic_5_no_sampling_settings_pass_through_silently(+ bedrock_provider: BedrockProvider, recwarn: pytest.WarningsRecorder+):+ """A flagged model whose settings carry no sampling parameters is left alone, with no warning."""+ model = BedrockConverseModel('eu.anthropic.claude-opus-5', provider=bedrock_provider)++ prepared, _ = model.prepare_request(BedrockModelSettings(max_tokens=16), ModelRequestParameters())++ assert prepared == snapshot({'max_tokens': 16})+ assert not [w for w in recwarn if 'Sampling parameters' in str(w.message)]
What the agent reported
I'll resume when the test gate completes.