Backtestspydanticrun-20260929-204949#13802

Issue #13802 Merged fix PR #13821 Run 29 Sep 2026 · run-20260929-204949 pydantic/pydantic

Three error-reference examples no longer raise the error they document, and the docs test suite cannot detect it

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches fix complex_type identically (False -> 'not a complex number') and delete the stale string_sub_type docs section. The decisive differences are scope/risk and tests. Patch A reaches across the package boundary into pydantic-core, deleting the StringSubType variant from types.rs, core_schema.py's ErrorType, and test_errors.py. That is a breaking API change to a separate compiled library (Rust), unwarranted by a docs-fix issue, and it removes an error type that pydantic-core still legitimately declares — high risk for little benefit; the issue explicitly left this as 'your call.' Patch B instead keeps the variant and simply excludes it in test_validation_error_codes with a clear explanatory comment, which is the lower-risk way to keep the code/docs consistency check green while matching reality (the variant still exists in core). On tests, Patch B directly implements the regression guard the issue asked for as its main point: a new test that AST-injects a sentinel into every except branch of the error-reference examples and fails loudly if any branch is never reached — precisely the blind spot described, and a genuine fail-before/pass-after test. Patch A only removes a line from an existing test, adding no protection against recurrence. Patch B's new test also self-validates its own invalid-for-json-schema replacement (EllipsisType) by requiring that example to reach its except branch. Patch A's Callable choice is more canonically known to lack a JSON schema, a slight correctness edge, but not enough to outweigh Patch B's superior scope discipline and the meaningful regression test the issue requested.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

22m 27s31 turns

Wall-clock time in the container

Full test suite

No new failures

12,723 passed · 526 skipped · 13,249 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

93vs 39 merged

Changed lines, ours vs merged

Initial Checks

  • I confirm that I'm using Pydantic V2

Description

Three examples on the error-reference pages no longer raise the error they document, and the documentation test suite cannot detect it.

Every example on those pages has the same shape: do something that should fail, catch it, then print or assert the error code. The only assertion lives inside the except branch. When the operation stops failing, that branch never runs, so nothing is printed and no assert is evaluated. run_print_check compares printed output against the #> comments, and an example whose print never ran produces no output to disagree with, so the example exits 0 and CI stays green while the page documents behaviour that no longer exists.

I checked this exhaustively rather than by sampling. I injected a sentinel as the first statement of every except block on both error pages and ran each example. Of the 152 examples containing a try/except at v2.13.5, all 152 run, and exactly 3 never enter their except branch. On main the count is 3 of 157. None of the three is marked test="skip", so all three are executed by CI today.

1. complex_type, docs/errors/validation_errors.md

Model(num=False) now validates successfully to 0j, so the #> 'complex_type' line never prints. This changed in 2.13.0b1 with #12498, which made complex validation call the complex() constructor unconditionally in Python mode, and complex(False) is 0j.

2. string_sub_type, docs/errors/validation_errors.md

The section says a strict str field rejects an instance of a str subclass. It does not: the example returns a plain str with the value 'foo', and #> 'string_sub_type' never prints. pydantic_core 2.49.0 still declares the error ("Input should be a string, not an instance of a subclass of str"), but I could not find any str schema that raises it. Building core_schema.str_schema() directly with strict=True, without strict, and with strict=True plus a pattern all accept a str subclass and return a plain str.

This looks intentional rather than accidental. #6261 reported exactly this behaviour as a bug in June 2023, was labelled bug V2, and was closed with the conversation moved to pydantic-core#786. So the check appears to have been removed on purpose and this documentation section was left behind describing it.

3. invalid-for-json-schema, docs/errors/usage_errors.md

ImportString has had a JSON schema since 2.8.0 via #9344, so Model.model_json_schema() succeeds and returns a plain string schema. The assert exc_info.code == 'invalid-for-json-schema' inside the except block has not executed since that release.

Output of the example code below

Identical on 2.13.5 and on main at 2.14.0b2:

1. no exception; num = 0j
2. no exception; accepted 'foo'
3. no exception; {'properties': {'a': {'title': 'A', 'type': 'string'}}, 'required': ['a'], 'title': 'ImportStringModel', 'type': 'object'}

Suggested fixes

  1. complex_type is a one-word fix. Model(num='not a complex number') still raises complex_type, and so do [], {} and a bare object(). Replacing False with any of those restores the example.
  2. string_sub_type needs a decision rather than a fix, which is the main reason I am opening an issue instead of sending a patch. Given the trail through #6261, the section probably wants deleting rather than repairing, and possibly the error variant too, but that is your call.
  3. invalid-for-json-schema needs a type that genuinely has no JSON schema, in place of ImportString.

Separately, the blind spot itself is fixable. If an example on these pages had to demonstrate that the error occurred, for instance by printing inside the except and also failing loudly when nothing was raised, then a future change like #12498 would turn the docs red instead of silently making a section untrue. I am happy to send a PR for any or all of this, including that check as a test, if you tell me which direction you prefer.

I used AI assistance to run this audit, in line with your AI policy. I have reproduced and understood each of the three cases myself and can explain any of them.

Example Code

from enum import Enum

from pydantic import (
    BaseModel,
    Field,
    ImportString,
    PydanticUserError,
    ValidationError,
)

# 1. validation_errors.md, `complex_type`: documented to print 'complex_type'
class ComplexModel(BaseModel):
    num: complex

try:
    ComplexModel(num=False)
    print('1. no exception; num =', ComplexModel(num=False).num)
except ValidationError as exc:
    print('1.', repr(exc.errors()[0]['type']))

# 2. validation_errors.md, `string_sub_type`: documented to print 'string_sub_type'
class MyEnum(str, Enum):
    foo = 'foo'

class StrictStrModel(BaseModel):
    x: str = Field(strict=True)

try:
    print('2. no exception; accepted', repr(StrictStrModel(x=MyEnum.foo).x))
except ValidationError as exc:
    print('2.', repr(exc.errors()[0]['type']))

# 3. usage_errors.md, `invalid-for-json-schema`: the assert is never reached
class ImportStringModel(BaseModel):
    a: ImportString

try:
    print('3. no exception;', ImportStringModel.model_json_schema())
except PydanticUserError as exc:
    print('3.', exc.code)

Python, Pydantic & OS Version

pydantic version: 2.13.5
        pydantic-core version: 2.46.5
        python version: 3.12.10 (CPython)
        platform: Windows 11

also reproduced against main at pydantic 2.14.0b2 / pydantic-core 2.49.0

The two patches

Our patch+65 −28
docs/errors/usage_errors.mdindex 7e28e9f8..3a9f3437 100644@@ -620,11 +620,13 @@ type C = B This error is raised when Pydantic fails to generate a JSON schema for some `CoreSchema`.  ```python-from pydantic import BaseModel, ImportString, PydanticUserError+from types import EllipsisType++from pydantic import BaseModel, PydanticUserError   class Model(BaseModel):-    a: ImportString+    a: EllipsisType   try:docs/errors/validation_errors.mdindex 71d3a929..95e240e7 100644@@ -233,7 +233,7 @@ class Model(BaseModel):   try:-    Model(num=False)+    Model(num='not a complex number') except ValidationError as exc:     print(repr(exc.errors()[0]['type']))     #> 'complex_type'@@ -1845,31 +1845,6 @@ except ValidationError as exc:     #> 'string_pattern_mismatch' ``` -## `string_sub_type`--This error is raised when the value is an instance of a strict subtype of `str` when the field is strict:--```python-from enum import Enum--from pydantic import BaseModel, Field, ValidationError---class MyEnum(str, Enum):-    foo = 'foo'---class Model(BaseModel):-    x: str = Field(strict=True)---try:-    Model(x=MyEnum.foo)-except ValidationError as exc:-    print(repr(exc.errors()[0]['type']))-    #> 'string_sub_type'-```- ## `string_too_long`  This error is raised when the input value is a string whose length is greater than the field's `max_length` constraint:tests/test_docs.pyindex 7c185657..846b9924 100644@@ -265,6 +265,7 @@ def test_validation_error_codes():     expected_validation_error_codes = set(core_schema.ErrorType.__args__)     # Remove codes that are not currently accessible from pydantic:     expected_validation_error_codes.remove('timezone_offset')  # not currently exposed for configuration in pydantic+    expected_validation_error_codes.remove('string_sub_type')  # defined in pydantic-core's ErrorType but never raised; strict str now accepts str subclasses      test_failures = [] @@ -291,3 +292,62 @@ def test_validation_error_codes():      code_validation_error_codes = sorted(expected_validation_error_codes)     assert code_validation_error_codes == documented_validation_error_codes, 'Error codes in code and docs do not match'+++def test_error_reference_examples_reach_except(eval_example: EvalExample, mocker: Any) -> None:+    """Every ``try``/``except`` example on the error-reference pages must enter its ``except`` branch.++    Each such example demonstrates an error and asserts it only inside ``except`` (by printing the+    error code, or via ``assert``). If the operation stops failing, the ``except`` branch never runs+    and the example silently passes while the page keeps documenting behaviour that no longer exists+    (the docs print-check cannot notice a ``print`` that never happened). This test runs each example+    the same way the docs runner does and fails loudly if any ``except`` branch is not reached.+    """+    if reason := skip_docs_tests():+        pytest.skip(reason)++    import ast++    error_pages = [DOCS_ROOT / 'errors' / 'validation_errors.md', DOCS_ROOT / 'errors' / 'usage_errors.md']+    mocker.patch('random.randint', return_value=3)++    failures: list[str] = []+    for example in find_examples(*[str(p) for p in error_pages]):+        prefix_settings = example.prefix_settings()+        if prefix_settings.get('test', '').startswith('skip'):+            continue+        requires = prefix_settings.get('requires')+        if requires:+            major, minor = map(int, requires.split('.'))+            if sys.version_info < (major, minor):+                continue++        tree = ast.parse(example.source)+        handlers = [node for node in ast.walk(tree) if isinstance(node, ast.ExceptHandler)]+        if not handlers:+            continue++        for idx, handler in enumerate(handlers):+            marker = ast.parse(f'pydantic_docs_except_reached.append({idx})').body[0]+            handler.body.insert(0, marker)+        ast.fix_missing_locations(tree)+        example.source = ast.unparse(tree) + '\n'++        reached: list[int] = []+        rewrite_assertions = prefix_settings.get('rewrite_assert', 'true') == 'true'+        with time_machine.travel(datetime(2032, 1, 2, 3, 4, 5, 6).astimezone(), tick=False):+            eval_example.run(+                example,+                module_globals={'pydantic_docs_except_reached': reached},+                rewrite_assertions=rewrite_assertions,+            )++        if set(reached) != set(range(len(handlers))):+            failures.append(+                f'{example.path.relative_to(DOCS_ROOT)}:{example.start_line}: an "except" branch never '+                f'executed; the example no longer raises the documented error'+            )++    assert not failures, 'error-reference examples whose "except" branch no longer runs:\n' + '\n'.join(+        f'  {f}' for f in failures+    ) 
Merged fix · PR #13821+6 −33
docs/concepts/experimental.mdindex cc349f3a0..ab1665260 100644@@ -342,7 +342,7 @@ class MyModel(BaseModel): ta = TypeAdapter(MyModel) try:     v = ta.validate_json(-        '{"a": 1, "b": ["12345", "12', experimental_allow_partial=True+        '{"a": 1, "b": ["12345", "12"', experimental_allow_partial=True     ) except ValidationError as e:     print(e)docs/errors/usage_errors.mdindex 7e28e9f82..3e6a67412 100644@@ -620,11 +620,13 @@ type C = B This error is raised when Pydantic fails to generate a JSON schema for some `CoreSchema`.  ```python-from pydantic import BaseModel, ImportString, PydanticUserError+from collections.abc import Callable++from pydantic import BaseModel, PydanticUserError   class Model(BaseModel):-    a: ImportString+    a: Callable[..., object]   try:docs/errors/validation_errors.mdindex 71d3a9292..95e240e70 100644@@ -233,7 +233,7 @@ class Model(BaseModel):   try:-    Model(num=False)+    Model(num='not a complex number') except ValidationError as exc:     print(repr(exc.errors()[0]['type']))     #> 'complex_type'@@ -1845,31 +1845,6 @@ except ValidationError as exc:     #> 'string_pattern_mismatch' ``` -## `string_sub_type`--This error is raised when the value is an instance of a strict subtype of `str` when the field is strict:--```python-from enum import Enum--from pydantic import BaseModel, Field, ValidationError---class MyEnum(str, Enum):-    foo = 'foo'---class Model(BaseModel):-    x: str = Field(strict=True)---try:-    Model(x=MyEnum.foo)-except ValidationError as exc:-    print(repr(exc.errors()[0]['type']))-    #> 'string_sub_type'-```- ## `string_too_long`  This error is raised when the input value is a string whose length is greater than the field's `max_length` constraint:pydantic-core/python/pydantic_core/core_schema.pyindex b6fce060b..fda0e35ee 100644@@ -4776,7 +4776,6 @@ ErrorType: TypeAlias = Literal[     'iterable_type',     'iteration_error',     'string_type',-    'string_sub_type',     'string_unicode',     'string_too_short',     'string_too_long',pydantic-core/src/errors/types.rsindex 4bdb92135..f49f48509 100644@@ -247,7 +247,6 @@ error_types! {     // ---------------------     // string errors     StringType {},-    StringSubType {},     StringUnicode {},     StringTooShort {         min_length: {ctx_type: usize, ctx_fn: field_from_context},@@ -533,7 +532,6 @@ impl ErrorType {             Self::IterableType { .. } => "Input should be iterable",             Self::IterationError { .. } => "Error iterating over object, error: {error}",             Self::StringType { .. } => "Input should be a valid string",-            Self::StringSubType { .. } => "Input should be a string, not an instance of a subclass of str",             Self::StringUnicode { .. } => {                 "Input should be a valid string, unable to parse raw data as a unicode string"             }pydantic-core/tests/test_errors.pyindex 66f15d81a..b024e618b 100644@@ -303,7 +303,6 @@ all_errors = [         {'field_type': 'Foobar', 'max_length': 42, 'actual_length': 50},     ),     ('string_type', 'Input should be a valid string', None),-    ('string_sub_type', 'Input should be a string, not an instance of a subclass of str', None),     ('string_unicode', 'Input should be a valid string, unable to parse raw data as a unicode string', None),     ('string_pattern_mismatch', "String should match pattern 'foo'", {'pattern': 'foo'}),     ('string_not_ascii', 'String should contain only ASCII characters', None), 

What the agent reported

All stages complete and verified. Here is the final report.

What was wrong

Three examples on the error-reference docs pages no longer raised the errors they documented, and the docs test suite couldn't detect it — each example's only assertion lives inside an except branch, so when the operation stops failing, the branch never runs, nothing prints, and CI stays green.

  • complex_type (validation_errors.md): Model(num=False) now validates to 0j instead of raising.
  • string_sub_type (validation_errors.md): strict str now accepts str subclasses. I confirmed in the pydantic-core Rust source that StringSubType is defined and given a message but never constructed anywhere — it's a dead variant, so no runnable example can demonstrate it.
  • invalid-for-json-schema (usage_errors.md): ImportString gained a JSON schema, so model_json_schema() succeeds and the except never runs.

What I changed (docs + tests only; no production/runtime code)

  1. docs/errors/validation_errors.md: Model(num=False) → Model(num='not a complex number') (still raises complex_type, verified against the Rust input path).
  2. docs/errors/validation_errors.md: deleted the string_sub_type section entirely (unrepairable dead variant).
  3. docs/errors/usage_errors.md: ImportString → EllipsisType (from types), which genuinely has no JSON schema and raises invalid-for-json-schema.
  4. tests/test_docs.py — test_validation_error_codes: added expected_validation_error_codes.remove('string_sub_type') with a comment, exactly mirroring the existing timezone_offset precedent. This was required because that test asserts every ErrorType member (minus inaccessible ones) is documented; deleting the section without it would break the test. This maintains, not weakens, the test.
  5. tests/test_docs.py — new regression test test_error_reference_examples_reach_except: the blind-spot detector the reporter proposed. It runs every try/except example on both error pages through the real docs runner (eval_example.run) with a sentinel injected into each except body, and fails loudly if any except branch is never reached.

Test evidence

  • Regression test fails before the fix with exactly the three reported examples (validation_errors.md:227, validation_errors.md:1852, usage_errors.md:622).
  • After the fix: full tests/test_docs.py → 621 passed, 108 skipped (normal), including test_docs_examples (lints + runs + print-checks every doc block, including the two corrected ones), test_validation_error_codes, test_error_codes, and the new regression test.
  • Reviewer confirmed all three cases + the optional test are correctly addressed, no unrelated/scratch changes, and no production code touched.

Requirements disposition

#Ask from reportDisposition
1Fix complex_type exampleImplemented — input changed; verified raises complex_type; passes test_docs_examples print-check.
2Decide on string_sub_type (delete vs repair)Implemented (deleted) — confirmed unrepairable (dead Rust variant, never constructed); section removed + expected-set updated. Error variant itself left in pydantic-core (out of scope; a separate Rust package change).
3Fix invalid-for-json-schema exampleImplemented — replaced with EllipsisType; verified raises code invalid-for-json-schema.
4Add a test so the blind spot fails loudly in futureImplemented — test_error_reference_examples_reach_except; verified fail-before / pass-after.

Reviewer noted one non-blocking design property worth recording: the new test verifies each except is entered (the exact #> code is still validated by the existing test_docs_examples print-check), and it assumes every except on these pages is meant to run — true today (matched try/except counts, no group=/xfail on these pages). All changes are left uncommitted in the working tree.