Backtestspydantic-airun-20260918-215936#5895
UsageLimits: add a per-request input-token (context size) limit
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches are behaviorally correct: each adds `per_request_input_tokens_limit`, enforces it against `count_tokens` output when `count_tokens_before_request=True` and against the response's provider-reported `input_tokens` otherwise, and neither accumulates it across requests. So correctness is essentially tied. The decision turns on tests and craft. Tests: The core requirement in the issue is the per-request-vs-cumulative distinction. Patch A's `test_per_request_input_tokens_limit_not_summed_across_requests` proves this deterministically with a FunctionModel emitting exactly 30 input tokens per request under a limit of 50, then asserts the run succeeds AND `result.usage.input_tokens == 60` — i.e. cumulative (60) exceeds the limit while each request (30) does not. This exactly discriminates a correct per-request check from a buggy cumulative one. Patch B's analogous `test_per_request_input_tokens_limit_multiple_requests` relies on TestModel's implicit token counts and asserts only `call_count == 2` and `result.output is not None`, which is weak and could pass even under some incorrect implementations; it doesn't pin down that the cumulative total would have tripped the limit. Patch B does add a nice standalone unit test for its helper and a good post-response snapshot test, so its suite has breadth, but its central semantic test is the weaker of the two. Craft: Patch A mirrors the existing error-message conventions the issue explicitly references — 'The next request would exceed the per_request_input_tokens_limit' for the pre-request path and 'Exceeded the per_request_input_tokens_limit' for the post-response path — matching how `input_tokens_limit` is worded in `check_before_request` vs `check_tokens`. Patch B uses a single 'Exceeded ...' message for both paths, diverging from that pre-request wording convention. Patch B's counterweight is genuinely good: a dedicated single-responsibility `check_per_request_input_tokens(int)` method that leaves the existing `check_before_request`/`check_tokens` signatures untouched, which is arguably cleaner scope-wise than A's threading of an optional `request_usage` param into two existing methods. That is a real point in B's favor. On balance, with correctness tied, Patch A's more rigorous discriminating test for the issue's central 'not summed' semantics (priority 2) plus its closer adherence to existing error-message conventions outweighs B's cleaner method separation. A wins narrowly.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
14m 58s1 turn
Wall-clock time in the container
Full test suite
No new failures
7,839 passed · 530 skipped · 8,403 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
114vs 138 merged
Changed lines, ours vs merged
The issue
View on GitHubDavid's AICA here:
UsageLimits currently offers only cumulative limits: request_limit, tool_calls_limit, input_tokens_limit, output_tokens_limit, total_tokens_limit (pydantic_ai_slim/pydantic_ai/usage.py, UsageLimits). There is no way to limit the size of any single request's input -- the context window actually sent to the model.
With prompt caching now standard across providers (implicit on DeepSeek/GLM/Moonshot, explicit on Anthropic), cumulative input tokens is a poor guard:
- Re-sent cached prefixes are cheap (e.g. DeepSeek bills cache reads at ~0.8% of input price), so a cumulative input cap mostly counts money that was never spent.
- The thing that does hurt -- cost on cache misses and degraded model performance on very long contexts -- is per-request context size, which no limit covers.
Real-world failure that motivated this: our dogfooding PR-opening agent (Pydanty, running on team.pydantic.work) died with UsageLimitExceeded: Exceeded the input_tokens_limit of 500000 (input_tokens=517132) after 26 requests averaging ~20k context each. No single request was over ~30k tokens; the run was healthy and ~90% complete when the cumulative counter killed it. The limit we actually wanted to express was "no single request may exceed N tokens", with the cumulative cap relegated to a generous catastrophic ceiling.
Proposal: add per_request_input_tokens_limit: int | None = None (name open) to UsageLimits, checked against each request's input token count -- ahead of the request when count_tokens_before_request=True, otherwise against the provider-reported input_tokens of the response, mirroring how the cumulative limits are enforced today.
This also gives history processors / compaction strategies a principled trigger to align with: compaction keeps you under the per-request limit; the cumulative limits stay what they are good at (runaway-loop ceilings).
The two patches
pydantic_ai_slim/pydantic_ai/_agent_graph.pyindex f3839d64..8bca0fdd 100644@@ -1016,14 +1016,16 @@ class ModelRequestNode(AgentNode[DepsT, NodeRunEndT]): ctx.state.last_max_tokens = model_settings.get('max_tokens') if model_settings else None ctx.state.last_model_request_parameters = model_request_parameters usage = ctx.state.usage+ request_usage = None if ctx.deps.usage_limits.count_tokens_before_request: # Copy to avoid modifying the original usage object with the counted usage usage = deepcopy(usage) counted_usage = await model.count_tokens(messages, model_settings, model_request_parameters) usage.incr(counted_usage)+ request_usage = counted_usage - ctx.deps.usage_limits.check_before_request(usage)+ ctx.deps.usage_limits.check_before_request(usage, request_usage) return model, model_settings or None, model_request_parameters, messages, run_context @@ -1087,7 +1089,7 @@ class ModelRequestNode(AgentNode[DepsT, NodeRunEndT]): fill_run_metadata(response, run_id=ctx.state.run_id, conversation_id=ctx.state.conversation_id) ctx.state.usage.incr(response.usage) if ctx.deps.usage_limits: # pragma: no branch- ctx.deps.usage_limits.check_tokens(ctx.state.usage)+ ctx.deps.usage_limits.check_tokens(ctx.state.usage, response.usage) ctx.state.message_history.append(response) async def _build_retry_node(pydantic_ai_slim/pydantic_ai/usage.pyindex d4ed3746..4401bf59 100644@@ -275,6 +275,18 @@ class UsageLimits: """The maximum number of output/response tokens allowed.""" total_tokens_limit: int | None = None """The maximum number of tokens allowed in requests and responses combined."""+ per_request_input_tokens_limit: int | None = None+ """The maximum number of input/prompt tokens allowed in any single request's context.++ Unlike `input_tokens_limit`, which caps the cumulative input tokens across the whole run,+ this caps the input tokens of each individual request and is not summed across requests.+ This is a better fit when prompt caching makes re-sent context cheap: cumulative input is a poor+ proxy for cost, whereas per-request context size is what drives cache-miss cost and long-context+ degradation.++ When `count_tokens_before_request=True`, this is enforced ahead of the request using the model's+ token-counting pass; otherwise it is enforced against the provider-reported input token count of each response.+ """ count_tokens_before_request: bool = False """If True, perform a token counting pass before sending the request to the model, to enforce `input_tokens_limit` ahead of time.@@ -302,7 +314,7 @@ class UsageLimits: limit is not None for limit in (self.input_tokens_limit, self.output_tokens_limit, self.total_tokens_limit) ) - def check_before_request(self, usage: RunUsage) -> None:+ def check_before_request(self, usage: RunUsage, request_usage: RequestUsage | None = None) -> None: """Raises a `UsageLimitExceeded` exception if the next request would exceed any of the limits.""" request_limit = self.request_limit if request_limit is not None and usage.requests >= request_limit:@@ -314,18 +326,34 @@ class UsageLimits: f'The next request would exceed the input_tokens_limit of {self.input_tokens_limit} ({input_tokens=})' ) + if request_usage is not None and self.per_request_input_tokens_limit is not None:+ request_input_tokens = request_usage.input_tokens+ if request_input_tokens > self.per_request_input_tokens_limit:+ raise UsageLimitExceeded(+ 'The next request would exceed the per_request_input_tokens_limit of '+ f'{self.per_request_input_tokens_limit} ({request_input_tokens=})'+ )+ total_tokens = usage.total_tokens if self.total_tokens_limit is not None and total_tokens > self.total_tokens_limit: raise UsageLimitExceeded( # pragma: lax no cover f'The next request would exceed the total_tokens_limit of {self.total_tokens_limit} ({total_tokens=})' ) - def check_tokens(self, usage: RunUsage) -> None:+ def check_tokens(self, usage: RunUsage, request_usage: RequestUsage | None = None) -> None: """Raises a `UsageLimitExceeded` exception if the usage exceeds any of the token limits.""" input_tokens = usage.input_tokens if self.input_tokens_limit is not None and input_tokens > self.input_tokens_limit: raise UsageLimitExceeded(f'Exceeded the input_tokens_limit of {self.input_tokens_limit} ({input_tokens=})') + if request_usage is not None and self.per_request_input_tokens_limit is not None:+ request_input_tokens = request_usage.input_tokens+ if request_input_tokens > self.per_request_input_tokens_limit:+ raise UsageLimitExceeded(+ f'Exceeded the per_request_input_tokens_limit of {self.per_request_input_tokens_limit} '+ f'({request_input_tokens=})'+ )+ output_tokens = usage.output_tokens if self.output_tokens_limit is not None and output_tokens > self.output_tokens_limit: raise UsageLimitExceeded(tests/test_usage_limits.pyindex f8dcbe3e..2bcca339 100644@@ -24,9 +24,11 @@ from pydantic_ai import ( UserPromptPart, ) from pydantic_ai.exceptions import ModelRetry+from pydantic_ai.models import ModelRequestParameters from pydantic_ai.models.function import AgentInfo, FunctionModel from pydantic_ai.models.test import TestModel from pydantic_ai.output import ToolOutput+from pydantic_ai.settings import ModelSettings from pydantic_ai.usage import RequestUsage, RunUsage, UsageLimits from ._inline_snapshot import snapshot@@ -993,3 +995,77 @@ def test_usage_limits_explicit_zero(): limits = UsageLimits(input_tokens_limit=100) assert limits.input_tokens_limit == 100+++async def test_per_request_input_tokens_limit_exceeded() -> None:+ """A single request whose input tokens exceed the per-request cap is rejected against the response."""++ def return_response(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:+ return ModelResponse(+ parts=[TextPart('hello')],+ usage=RequestUsage(input_tokens=60, output_tokens=5),+ )++ test_agent = Agent(FunctionModel(return_response))++ with pytest.raises(+ UsageLimitExceeded,+ match=re.escape('Exceeded the per_request_input_tokens_limit of 5 (request_input_tokens=60)'),+ ):+ await test_agent.run('test', usage_limits=UsageLimits(per_request_input_tokens_limit=5))+++async def test_per_request_input_tokens_limit_not_summed_across_requests() -> None:+ """The per-request cap applies to each request individually and is NOT accumulated across requests."""++ def two_requests(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:+ if len(messages) > 1:+ return ModelResponse(+ parts=[TextPart('done')],+ usage=RequestUsage(input_tokens=30, output_tokens=5),+ )+ return ModelResponse(+ parts=[ToolCallPart('regular_tool', {'x': 'test'}, 'call_1')],+ usage=RequestUsage(input_tokens=30, output_tokens=5),+ )++ test_agent = Agent(FunctionModel(two_requests))++ @test_agent.tool_plain+ async def regular_tool(x: str) -> str:+ return f'{x}-processed'++ result = await test_agent.run('test', usage_limits=UsageLimits(per_request_input_tokens_limit=50))++ assert result.output == 'done'+ assert result.usage.input_tokens == 60+++async def test_per_request_input_tokens_limit_before_request() -> None:+ """When counting tokens ahead of the request, the per-request cap is enforced before sending."""++ class CountTokensFunctionModel(FunctionModel):+ async def count_tokens(+ self,+ messages: list[ModelMessage],+ model_settings: ModelSettings | None,+ model_request_parameters: ModelRequestParameters,+ ) -> RequestUsage:+ return RequestUsage(input_tokens=60)++ def return_response(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:+ return ModelResponse(+ parts=[TextPart('hello')],+ usage=RequestUsage(input_tokens=60, output_tokens=5),+ )++ test_agent = Agent(CountTokensFunctionModel(return_response))++ with pytest.raises(+ UsageLimitExceeded,+ match=re.escape('The next request would exceed the per_request_input_tokens_limit of 5 (request_input_tokens=60)'),+ ):+ await test_agent.run(+ 'test',+ usage_limits=UsageLimits(per_request_input_tokens_limit=5, count_tokens_before_request=True),+ )
pydantic_ai_slim/pydantic_ai/_agent_graph.pyindex f3839d64e6..9f0435484c 100644@@ -1023,6 +1023,8 @@ class ModelRequestNode(AgentNode[DepsT, NodeRunEndT]): counted_usage = await model.count_tokens(messages, model_settings, model_request_parameters) usage.incr(counted_usage) + ctx.deps.usage_limits.check_per_request_input_tokens(counted_usage.input_tokens)+ ctx.deps.usage_limits.check_before_request(usage) return model, model_settings or None, model_request_parameters, messages, run_context@@ -1088,6 +1090,7 @@ class ModelRequestNode(AgentNode[DepsT, NodeRunEndT]): ctx.state.usage.incr(response.usage) if ctx.deps.usage_limits: # pragma: no branch ctx.deps.usage_limits.check_tokens(ctx.state.usage)+ ctx.deps.usage_limits.check_per_request_input_tokens(response.usage.input_tokens) ctx.state.message_history.append(response) async def _build_retry_node(pydantic_ai_slim/pydantic_ai/usage.pyindex d4ed374625..3136b8e844 100644@@ -275,6 +275,18 @@ class UsageLimits: """The maximum number of output/response tokens allowed.""" total_tokens_limit: int | None = None """The maximum number of tokens allowed in requests and responses combined."""+ per_request_input_tokens_limit: int | None = None+ """The maximum number of input/prompt tokens allowed per individual request.++ Unlike `input_tokens_limit` which is cumulative across the entire run, this+ limit is checked against each request's input token count independently —+ ahead of the request when `count_tokens_before_request=True`, otherwise against+ the provider-reported `input_tokens` of the response.++ This provides a guard against oversized contexts (which hurt model performance+ and incur high costs on cache misses), complementing the runaway-loop+ protection that cumulative limits provide.+ """ count_tokens_before_request: bool = False """If True, perform a token counting pass before sending the request to the model, to enforce `input_tokens_limit` ahead of time.@@ -345,4 +357,16 @@ class UsageLimits: f'The next tool call(s) would exceed the tool_calls_limit of {tool_calls_limit} ({tool_calls=}).' ) + def check_per_request_input_tokens(self, request_input_tokens: int) -> None:+ """Raises a `UsageLimitExceeded` if the per-request input tokens exceed the limit.++ This checks a single request's input token count — not the cumulative+ `RunUsage.input_tokens` — against `per_request_input_tokens_limit`.+ """+ limit = self.per_request_input_tokens_limit+ if limit is not None and request_input_tokens > limit:+ raise UsageLimitExceeded(+ f'Exceeded the per_request_input_tokens_limit of {limit} ({request_input_tokens=})'+ )+ __repr__ = _utils.dataclasses_no_defaults_reprtests/test_usage_limits.pyindex f8dcbe3e0e..a498f9f2c6 100644@@ -5,6 +5,7 @@ import re from collections.abc import AsyncIterator from datetime import timezone from decimal import Decimal+from typing import Any import pytest from genai_prices import Usage as GenaiPricesUsage, calc_price@@ -993,3 +994,113 @@ def test_usage_limits_explicit_zero(): limits = UsageLimits(input_tokens_limit=100) assert limits.input_tokens_limit == 100+++# ── per_request_input_tokens_limit ──────────────────────────────────────+++def test_per_request_input_tokens_limit_post_response() -> None:+ """When count_tokens_before_request=False (default), the limit is checked+ against the provider-reported input_tokens after the response."""+ test_agent = Agent(TestModel())++ with pytest.raises(+ UsageLimitExceeded,+ match=re.escape('Exceeded the per_request_input_tokens_limit of 5 (request_input_tokens=51)'),+ ):+ test_agent.run_sync('Hello', usage_limits=UsageLimits(per_request_input_tokens_limit=5))+++def test_per_request_input_tokens_limit_not_exceeded() -> None:+ """When per-request input tokens are below the limit, no error is raised."""+ test_agent = Agent(TestModel())++ result = test_agent.run_sync('Hello', usage_limits=UsageLimits(per_request_input_tokens_limit=100))+ assert result.output == 'success (no tool calls)'+++def test_per_request_input_tokens_limit_multiple_requests() -> None:+ """The limit is per-request, not cumulative.++ Each request's input tokens are checked independently, so a run with+ many small requests (each under the per-request limit) succeeds even+ though the cumulative total would exceed the per-request limit.+ """++ call_count = 0++ async def tool_a(ctx: RunContext[None]) -> str:+ nonlocal call_count+ call_count += 1+ return 'done'++ async def tool_b(ctx: RunContext[None]) -> str:+ nonlocal call_count+ call_count += 1+ return 'done'++ agent = Agent(+ TestModel(call_tools=['tool_a', 'tool_b']),+ tools=[tool_a, tool_b],+ )++ result = agent.run_sync(+ 'run tools',+ usage_limits=UsageLimits(per_request_input_tokens_limit=100),+ )+ assert call_count == 2+ # Each request had modest input tokens, under the 100 limit.+ # The cumulative total does not trigger the per-request limit.+ assert result.output is not None+++def test_check_per_request_input_tokens_unit() -> None:+ """Unit test for the check_per_request_input_tokens method."""+ limits = UsageLimits(per_request_input_tokens_limit=10)++ # Under limit — no error+ limits.check_per_request_input_tokens(5)+ limits.check_per_request_input_tokens(10)++ # Over limit — raises+ with pytest.raises(+ UsageLimitExceeded,+ match=re.escape('Exceeded the per_request_input_tokens_limit of 10 (request_input_tokens=11)'),+ ):+ limits.check_per_request_input_tokens(11)++ # None limit — never raises+ unlimited = UsageLimits(per_request_input_tokens_limit=None)+ unlimited.check_per_request_input_tokens(1_000_000)+++def test_per_request_input_tokens_limit_with_count_before() -> None:+ """When count_tokens_before_request=True and the counted tokens exceed the+ per-request limit, the error is raised before the model call."""++ counted = 200++ # TestModel doesn't implement count_tokens, so we patch it on the instance.+ async def fake_count_tokens(+ messages: list[ModelMessage],+ model_settings: Any,+ model_request_parameters: Any,+ ) -> RequestUsage:+ return RequestUsage(input_tokens=counted)++ model = TestModel()+ model.count_tokens = fake_count_tokens # type: ignore[method-assign]++ agent = Agent(model)++ with pytest.raises(+ UsageLimitExceeded,+ match=re.escape(f'Exceeded the per_request_input_tokens_limit of 10 (request_input_tokens={counted})'),+ ):+ agent.run_sync(+ 'Hello',+ usage_limits=UsageLimits(+ per_request_input_tokens_limit=10,+ count_tokens_before_request=True,+ ),+ )
What the agent reported
The reviewer confirms the implementation is correct with no functional gaps. The unrelated D deletions (.claude/settings.json, AGENTS.md, CLAUDE.md) were pre-existing at session start and are not part of my change — I left them untouched. The work is complete.
Summary
What was wrong
UsageLimits only offered cumulative token limits (input_tokens_limit, total_tokens_limit, etc.). With prompt caching now standard, cumulative input is a poor cost/health proxy — re-sent cached prefixes are cheap, so a cumulative cap counts money never spent and kills otherwise-healthy runs (the motivating failure: 26 healthy requests averaging ~20k context each tripped a 500k cumulative input_tokens_limit, even though no single request exceeded ~30k). There was no way to express "no single request may exceed N context tokens".
What I changed
pydantic_ai_slim/pydantic_ai/usage.py
- Added
per_request_input_tokens_limit: int | None = NonetoUsageLimits, with a docstring contrasting it against the cumulativeinput_tokens_limitand documenting both enforcement paths. check_before_request(usage, request_usage=None)— new optionalrequest_usageparam; enforces the per-request cap against the counted request'sinput_tokenswhen counting ahead ("The next request would exceed…").check_tokens(usage, request_usage=None)— new optional param; enforces the cap against a single response'sinput_tokens("Exceeded…"). Both checks read only the single-request usage, never the cumulativeRunUsage.
pydantic_ai_slim/pydantic_ai/_agent_graph.py
- Pre-request path passes
counted_usage(frommodel.count_tokens) tocheck_before_request. _append_responsepassesresponse.usagetocheck_tokens.- Streaming (
result.py) left unchanged: it callscheck_tokenswithoutrequest_usage, so per-request enforcement defers to the final response append (which always runs) — no mid-stream false accumulation, no slip-through.
Test evidence
Three new tests in tests/test_usage_limits.py, all failing before (TypeError: unexpected keyword argument 'per_request_input_tokens_limit') and passing after:
test_per_request_input_tokens_limit_exceeded— post-response path: input=60 vs cap 5 →UsageLimitExceeded: Exceeded the per_request_input_tokens_limit of 5 (request_input_tokens=60).test_per_request_input_tokens_limit_not_summed_across_requests— the motivating scenario: two 30-token requests under a cap of 50 succeed with cumulativeinput_tokens == 60, proving per-request is not accumulated.test_per_request_input_tokens_limit_before_request— count-ahead path:count_tokens_before_request=True→The next request would exceed the per_request_input_tokens_limit of 5 (request_input_tokens=60).
Gate: tests/test_usage_limits.py + test_anthropic_bedrock_count_tokens.py → 33 passed; blast-radius check on test_bedrock.py/test_openai_responses.py count-tokens/usage-limit tests → 12 passed. No existing test modified.
Requirements disposition
- Add
per_request_input_tokens_limit: int | None = None(name open) toUsageLimits— ✅ Implemented; field added with docstring. - Check against each request's input token count, ahead of the request when
count_tokens_before_request=True— ✅ Implemented incheck_before_request; covered bytest_per_request_input_tokens_limit_before_request. - Otherwise check against the provider-reported
input_tokensof the response — ✅ Implemented incheck_tokensviaresponse.usage; covered bytest_per_request_input_tokens_limit_exceeded. - Per-request (not cumulative) semantics — mirror how cumulative limits are enforced — ✅ Checks read only single-request usage;
test_per_request_input_tokens_limit_not_summed_across_requestsproves non-accumulation. - History-processor/compaction trigger alignment — informational framing in the report, not a code deliverable; the field is the principled trigger and needs no separate change. No code touched.
Note: I intentionally left has_token_limits() unchanged — per-request violations on a streamed response surface at stream completion (via _append_response) rather than mid-stream, since a single request's input can't be isolated from cumulative RunUsage during streaming. All changes are left uncommitted in the working tree.