Backtestspydanticrun-20260930-144322#13802
Three error-reference examples no longer raise the error they document, and the docs test suite cannot detect it
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the same three doc fixes. `complex_type` now uses a string input. The `string_sub_type` section is deleted. `invalid-for-json-schema` now uses a `Callable` field, which really has no JSON schema, so the `except` branch runs again. On those three fixes the patches are equivalent. They differ elsewhere. Patch A also deletes the `StringSubType` variant from pydantic-core's Rust enum, the Python `ErrorType` Literal, and the core error test. That removes part of the public error-type surface, which the issue explicitly left as the maintainers' decision. Patch A also edits `docs/concepts/experimental.md`, changing the partial-JSON example input from an unterminated string `"12` to a terminated `"12"`. That change is unrelated to the issue and may alter what the example is meant to show about partial validation. Both are risky scope changes. Patch A also adds no test for the blind spot the issue emphasises, so a future regression like the `complex` one would still pass silently. Patch B keeps the core error type. It updates `test_validation_error_codes` to exclude `string_sub_type`, with a comment explaining it is never raised; this is needed because the doc section is gone. It also adds `test_docs_error_examples_raise`. That test parses each error-page example and injects an `else: raise AssertionError` into every module-level `try`/`except` that has no `else`. It then executes the example, so any example that stops raising fails loudly. This test would have caught all three reported cases (it fails before the doc fixes and passes after), and it directly addresses the root cause the issue describes. It also respects the existing skip and requires settings. Patch B therefore fixes the issue more completely, with real regression tests and less risky churn.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
4m 6s12 turns
Wall-clock time in the container
Full test suite
No new failures
12,875 passed · 545 skipped · 13,420 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
85vs 39 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I confirm that I'm using Pydantic V2
Description
Three examples on the error-reference pages no longer raise the error they document, and the documentation test suite cannot detect it.
Every example on those pages has the same shape: do something that should fail, catch it, then print or assert the error code. The only assertion lives inside the except branch. When the operation stops failing, that branch never runs, so nothing is printed and no assert is evaluated. run_print_check compares printed output against the #> comments, and an example whose print never ran produces no output to disagree with, so the example exits 0 and CI stays green while the page documents behaviour that no longer exists.
I checked this exhaustively rather than by sampling. I injected a sentinel as the first statement of every except block on both error pages and ran each example. Of the 152 examples containing a try/except at v2.13.5, all 152 run, and exactly 3 never enter their except branch. On main the count is 3 of 157. None of the three is marked test="skip", so all three are executed by CI today.
1. complex_type, docs/errors/validation_errors.md
Model(num=False) now validates successfully to 0j, so the #> 'complex_type' line never prints. This changed in 2.13.0b1 with #12498, which made complex validation call the complex() constructor unconditionally in Python mode, and complex(False) is 0j.
2. string_sub_type, docs/errors/validation_errors.md
The section says a strict str field rejects an instance of a str subclass. It does not: the example returns a plain str with the value 'foo', and #> 'string_sub_type' never prints. pydantic_core 2.49.0 still declares the error ("Input should be a string, not an instance of a subclass of str"), but I could not find any str schema that raises it. Building core_schema.str_schema() directly with strict=True, without strict, and with strict=True plus a pattern all accept a str subclass and return a plain str.
This looks intentional rather than accidental. #6261 reported exactly this behaviour as a bug in June 2023, was labelled bug V2, and was closed with the conversation moved to pydantic-core#786. So the check appears to have been removed on purpose and this documentation section was left behind describing it.
3. invalid-for-json-schema, docs/errors/usage_errors.md
ImportString has had a JSON schema since 2.8.0 via #9344, so Model.model_json_schema() succeeds and returns a plain string schema. The assert exc_info.code == 'invalid-for-json-schema' inside the except block has not executed since that release.
Output of the example code below
Identical on 2.13.5 and on main at 2.14.0b2:
1. no exception; num = 0j
2. no exception; accepted 'foo'
3. no exception; {'properties': {'a': {'title': 'A', 'type': 'string'}}, 'required': ['a'], 'title': 'ImportStringModel', 'type': 'object'}
Suggested fixes
complex_typeis a one-word fix.Model(num='not a complex number')still raisescomplex_type, and so do[],{}and a bareobject(). ReplacingFalsewith any of those restores the example.string_sub_typeneeds a decision rather than a fix, which is the main reason I am opening an issue instead of sending a patch. Given the trail through #6261, the section probably wants deleting rather than repairing, and possibly the error variant too, but that is your call.invalid-for-json-schemaneeds a type that genuinely has no JSON schema, in place ofImportString.
Separately, the blind spot itself is fixable. If an example on these pages had to demonstrate that the error occurred, for instance by printing inside the except and also failing loudly when nothing was raised, then a future change like #12498 would turn the docs red instead of silently making a section untrue. I am happy to send a PR for any or all of this, including that check as a test, if you tell me which direction you prefer.
I used AI assistance to run this audit, in line with your AI policy. I have reproduced and understood each of the three cases myself and can explain any of them.
Example Code
from enum import Enum
from pydantic import (
BaseModel,
Field,
ImportString,
PydanticUserError,
ValidationError,
)
# 1. validation_errors.md, `complex_type`: documented to print 'complex_type'
class ComplexModel(BaseModel):
num: complex
try:
ComplexModel(num=False)
print('1. no exception; num =', ComplexModel(num=False).num)
except ValidationError as exc:
print('1.', repr(exc.errors()[0]['type']))
# 2. validation_errors.md, `string_sub_type`: documented to print 'string_sub_type'
class MyEnum(str, Enum):
foo = 'foo'
class StrictStrModel(BaseModel):
x: str = Field(strict=True)
try:
print('2. no exception; accepted', repr(StrictStrModel(x=MyEnum.foo).x))
except ValidationError as exc:
print('2.', repr(exc.errors()[0]['type']))
# 3. usage_errors.md, `invalid-for-json-schema`: the assert is never reached
class ImportStringModel(BaseModel):
a: ImportString
try:
print('3. no exception;', ImportStringModel.model_json_schema())
except PydanticUserError as exc:
print('3.', exc.code)
Python, Pydantic & OS Version
pydantic version: 2.13.5
pydantic-core version: 2.46.5
python version: 3.12.10 (CPython)
platform: Windows 11
also reproduced against main at pydantic 2.14.0b2 / pydantic-core 2.49.0The two patches
docs/errors/usage_errors.mdindex 7e28e9f8..abbdd276 100644@@ -620,11 +620,13 @@ type C = B This error is raised when Pydantic fails to generate a JSON schema for some `CoreSchema`. ```python-from pydantic import BaseModel, ImportString, PydanticUserError+from collections.abc import Callable++from pydantic import BaseModel, PydanticUserError class Model(BaseModel):- a: ImportString+ a: Callable[[], int] try:docs/errors/validation_errors.mdindex 71d3a929..95e240e7 100644@@ -233,7 +233,7 @@ class Model(BaseModel): try:- Model(num=False)+ Model(num='not a complex number') except ValidationError as exc: print(repr(exc.errors()[0]['type'])) #> 'complex_type'@@ -1845,31 +1845,6 @@ except ValidationError as exc: #> 'string_pattern_mismatch' ``` -## `string_sub_type`--This error is raised when the value is an instance of a strict subtype of `str` when the field is strict:--```python-from enum import Enum--from pydantic import BaseModel, Field, ValidationError---class MyEnum(str, Enum):- foo = 'foo'---class Model(BaseModel):- x: str = Field(strict=True)---try:- Model(x=MyEnum.foo)-except ValidationError as exc:- print(repr(exc.errors()[0]['type']))- #> 'string_sub_type'-```- ## `string_too_long` This error is raised when the input value is a string whose length is greater than the field's `max_length` constraint:tests/test_docs.pyindex 7c185657..75223504 100644@@ -1,10 +1,14 @@ from __future__ import annotations as _annotations +import ast+import contextlib+import io import os import platform import re import subprocess import sys+import types from datetime import datetime from pathlib import Path from tempfile import NamedTemporaryFile@@ -207,6 +211,53 @@ def test_docs_examples(example: CodeExample, eval_example: EvalExample, tmp_path run_example(example, eval_example, mocker) +@pytest.mark.thread_unsafe+@pytest.mark.filterwarnings('ignore:(parse_obj_as|schema_json_of|schema_of) is deprecated.*:DeprecationWarning')+@pytest.mark.skipif(bool(skip_reason), reason=skip_reason or 'not skipping')+@pytest.mark.parametrize(+ 'example',+ list(+ find_examples(+ str(DOCS_ROOT / 'errors/validation_errors.md'),+ str(DOCS_ROOT / 'errors/usage_errors.md'),+ skip=sys.platform == 'win32',+ )+ ),+ ids=str,+)+def test_docs_error_examples_raise(example: CodeExample):+ """Each module-level `try`/`except` in the error docs must actually hit its `except` branch.++ Otherwise an example that stops raising would silently pass `test_docs_examples`.+ """+ prefix_settings = example.prefix_settings()+ if prefix_settings.get('test', '').startswith('skip'):+ pytest.skip('running code skipped')+ requires_settings = prefix_settings.get('requires')+ if requires_settings and sys.version_info < tuple(map(int, requires_settings.split('.'))):+ pytest.skip(f'requires python {requires_settings}')++ tree = ast.parse(example.source)+ try_nodes = [node for node in tree.body if isinstance(node, ast.Try) and node.handlers]+ if not try_nodes:+ pytest.skip('no module-level try/except')++ for node in try_nodes:+ if not node.orelse:+ msg = f'no exception raised by the try block at line {node.lineno}'+ node.orelse = ast.parse(f'raise AssertionError({msg!r})').body+ ast.fix_missing_locations(tree)++ module_name = f'_docs_error_example_{example.path.stem}_{example.start_line}'+ module = types.ModuleType(module_name)+ sys.modules[module_name] = module+ try:+ with contextlib.redirect_stdout(io.StringIO()):+ exec(compile(tree, str(example.path), 'exec'), module.__dict__)+ finally:+ del sys.modules[module_name]++ @pytest.mark.thread_unsafe @pytest.mark.skipif(bool(skip_reason), reason=skip_reason or 'not skipping') @pytest.mark.skipif(sys.version_info >= (3, 13), reason='python-devtools does not yet support python 3.13')@@ -265,6 +316,7 @@ def test_validation_error_codes(): expected_validation_error_codes = set(core_schema.ErrorType.__args__) # Remove codes that are not currently accessible from pydantic: expected_validation_error_codes.remove('timezone_offset') # not currently exposed for configuration in pydantic+ expected_validation_error_codes.remove('string_sub_type') # never raised: str subclasses are accepted and coerced to str test_failures = []
docs/concepts/experimental.mdindex cc349f3a0..ab1665260 100644@@ -342,7 +342,7 @@ class MyModel(BaseModel): ta = TypeAdapter(MyModel) try: v = ta.validate_json(- '{"a": 1, "b": ["12345", "12', experimental_allow_partial=True+ '{"a": 1, "b": ["12345", "12"', experimental_allow_partial=True ) except ValidationError as e: print(e)docs/errors/usage_errors.mdindex 7e28e9f82..3e6a67412 100644@@ -620,11 +620,13 @@ type C = B This error is raised when Pydantic fails to generate a JSON schema for some `CoreSchema`. ```python-from pydantic import BaseModel, ImportString, PydanticUserError+from collections.abc import Callable++from pydantic import BaseModel, PydanticUserError class Model(BaseModel):- a: ImportString+ a: Callable[..., object] try:docs/errors/validation_errors.mdindex 71d3a9292..95e240e70 100644@@ -233,7 +233,7 @@ class Model(BaseModel): try:- Model(num=False)+ Model(num='not a complex number') except ValidationError as exc: print(repr(exc.errors()[0]['type'])) #> 'complex_type'@@ -1845,31 +1845,6 @@ except ValidationError as exc: #> 'string_pattern_mismatch' ``` -## `string_sub_type`--This error is raised when the value is an instance of a strict subtype of `str` when the field is strict:--```python-from enum import Enum--from pydantic import BaseModel, Field, ValidationError---class MyEnum(str, Enum):- foo = 'foo'---class Model(BaseModel):- x: str = Field(strict=True)---try:- Model(x=MyEnum.foo)-except ValidationError as exc:- print(repr(exc.errors()[0]['type']))- #> 'string_sub_type'-```- ## `string_too_long` This error is raised when the input value is a string whose length is greater than the field's `max_length` constraint:pydantic-core/python/pydantic_core/core_schema.pyindex b6fce060b..fda0e35ee 100644@@ -4776,7 +4776,6 @@ ErrorType: TypeAlias = Literal[ 'iterable_type', 'iteration_error', 'string_type',- 'string_sub_type', 'string_unicode', 'string_too_short', 'string_too_long',pydantic-core/src/errors/types.rsindex 4bdb92135..f49f48509 100644@@ -247,7 +247,6 @@ error_types! { // --------------------- // string errors StringType {},- StringSubType {}, StringUnicode {}, StringTooShort { min_length: {ctx_type: usize, ctx_fn: field_from_context},@@ -533,7 +532,6 @@ impl ErrorType { Self::IterableType { .. } => "Input should be iterable", Self::IterationError { .. } => "Error iterating over object, error: {error}", Self::StringType { .. } => "Input should be a valid string",- Self::StringSubType { .. } => "Input should be a string, not an instance of a subclass of str", Self::StringUnicode { .. } => { "Input should be a valid string, unable to parse raw data as a unicode string" }pydantic-core/tests/test_errors.pyindex 66f15d81a..b024e618b 100644@@ -303,7 +303,6 @@ all_errors = [ {'field_type': 'Foobar', 'max_length': 42, 'actual_length': 50}, ), ('string_type', 'Input should be a valid string', None),- ('string_sub_type', 'Input should be a string, not an instance of a subclass of str', None), ('string_unicode', 'Input should be a valid string, unable to parse raw data as a unicode string', None), ('string_pattern_mismatch', "String should match pattern 'foo'", {'pattern': 'foo'}), ('string_not_ascii', 'String should contain only ASCII characters', None),
What the agent reported
All three examples now raise the error they document, and a new docs test fails if an example on either error page stops raising. The docs tests for the error pages pass: 321 passed, 29 skipped, 0 failed. Nothing is committed; the changes are in docs/errors/validation_errors.md, docs/errors/usage_errors.md and tests/test_docs.py.
What was wrong: Model(num=False) now validates to 0j. A strict str field accepts str subclasses, so nothing raises string_sub_type any more. ImportString has had a JSON schema since 2.8. In each case the only check sat inside the except block, so when nothing was raised the example printed nothing and still passed.
Changes:
complex_type: the example now usesModel(num='not a complex number'). I checked that it raisescomplex_type.string_sub_type: I deleted the section, following #6261 and pydantic-core#786. The existingtest_validation_error_codesexpects everyErrorTypeto have a section in the docs. So I addedstring_sub_typeto its exclusion list, next to the existingtimezone_offsetexclusion, with a comment explaining why.- This is a change to an existing test, and it makes that check skip one error code.
- The only alternative was to keep an example that wrongly claims to print
'string_sub_type'. - I didn't remove the error variant from pydantic-core: that needs a Rust change and a rebuild, and I've left it for the maintainers to decide.
invalid-for-json-schema: the example now usesa: Callable[[], int](fromcollections.abc). I checked that it raises the error with this code.- The detection gap: I added
test_docs_error_examples_raisetotests/test_docs.py. For every example on the two error pages, it addselse: raise AssertionError(...)to each top-leveltry/exceptand runs the example. It follows the sametest="skip"andrequires=settings as the existing docs test.
Test results:
- The new test on its own: 153 passed, 19 skipped.
- With the docs temporarily put back to
num=FalseandImportString, it failed on exactly those two examples ("no exception raised by the try block"), so it catches the reported problem. ./bt-test tests/test_docs.py -k "error_codes or errors": 321 passed, 29 skipped, 0 failed.- An earlier run caught a lint error (UP035) on
from typing import Callable; switching tocollections.abcfixed it.
Limits of the new test: it only checks try blocks at the top level of an example, not ones inside functions. It only proves that the caught exception type was raised somewhere in the try, not which statement raised it.
Requirements:
| Item | Outcome |
|---|---|
| Fix the complex_type example | Done; covered by the new test and the docs example test |
| Decide on string_sub_type | Section deleted; exclusion added to the existing test; error variant left in pydantic-core |
| Replace ImportString in the invalid-for-json-schema example | Done, using Callable; covered by the new test |
| Make a silently passing error example fail the docs tests | Done; shown to fail on the old docs and pass on the new ones |
A reviewer read the diff and found no blocking issues or unrelated changes. It said the new test lacks the skipif marker the other docs tests have; I checked, and the marker is there.