Backtestspydanticrun-20260923-170312#13802

Issue #13802 Merged fix PR # Run 23 Sep 2026 · run-20260923-170312 pydantic/pydantic

Three error-reference examples no longer raise the error they document, and the docs test suite cannot detect it

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches apply the identical, correct documentation fixes: swapping `Model(num=False)` for a value that still raises `complex_type`, deleting the obsolete `string_sub_type` section, and replacing `ImportString` (which now has a JSON schema) with a `Callable` type that genuinely has none. They tie on the core doc corrections. The decision turns on tests and scope. Tests: Patch B adds a real, targeted regression test (`test_error_reference_examples_document_real_errors`) that pins down all three cases — it asserts that `num=False` now validates to 0j while the new input raises `complex_type`, that a str subclass is accepted and the removed section is gone, and that `ImportString` produces a schema while `Callable` raises `invalid-for-json-schema`. This would fail before the doc changes and pass after, and it directly answers the issue author's request for a guard against this blind spot. Patch A's only test change is deleting the `string_sub_type` row from pydantic-core's `all_errors` list — a mechanical consequence of removing the error variant, not a test of the reported behaviour. Scope: Patch A goes well beyond the docs issue by surgically removing the `StringSubType` variant from the Rust core (`types.rs`), the generated Python stub (`core_schema.py`), and the core test suite — a cross-package change to a separately published dependency that the issue explicitly flags as a discretionary 'your call' decision and that risks breaking the Rust build if any other reference remains (not visible in the diff). Patch A also includes an entirely unrelated drive-by edit to `docs/concepts/experimental.md` (a partial-JSON string), which is pure churn against this issue. Patch B keeps everything within the pydantic repo, achieves consistency by removing `string_sub_type` from the expected-codes set in `test_validation_error_codes` with an explanatory comment, and adds no incidental changes. Patch B is more correct at the appropriate scope, better tested, and cleaner.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

56m 49s47 turns

Wall-clock time in the container

Full test suite

No new failures

6,791 passed · 390 skipped · 7,181 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

109vs 39 merged

Changed lines, ours vs merged

Initial Checks

  • I confirm that I'm using Pydantic V2

Description

Three examples on the error-reference pages no longer raise the error they document, and the documentation test suite cannot detect it.

Every example on those pages has the same shape: do something that should fail, catch it, then print or assert the error code. The only assertion lives inside the except branch. When the operation stops failing, that branch never runs, so nothing is printed and no assert is evaluated. run_print_check compares printed output against the #> comments, and an example whose print never ran produces no output to disagree with, so the example exits 0 and CI stays green while the page documents behaviour that no longer exists.

I checked this exhaustively rather than by sampling. I injected a sentinel as the first statement of every except block on both error pages and ran each example. Of the 152 examples containing a try/except at v2.13.5, all 152 run, and exactly 3 never enter their except branch. On main the count is 3 of 157. None of the three is marked test="skip", so all three are executed by CI today.

1. complex_type, docs/errors/validation_errors.md

Model(num=False) now validates successfully to 0j, so the #> 'complex_type' line never prints. This changed in 2.13.0b1 with #12498, which made complex validation call the complex() constructor unconditionally in Python mode, and complex(False) is 0j.

2. string_sub_type, docs/errors/validation_errors.md

The section says a strict str field rejects an instance of a str subclass. It does not: the example returns a plain str with the value 'foo', and #> 'string_sub_type' never prints. pydantic_core 2.49.0 still declares the error ("Input should be a string, not an instance of a subclass of str"), but I could not find any str schema that raises it. Building core_schema.str_schema() directly with strict=True, without strict, and with strict=True plus a pattern all accept a str subclass and return a plain str.

This looks intentional rather than accidental. #6261 reported exactly this behaviour as a bug in June 2023, was labelled bug V2, and was closed with the conversation moved to pydantic-core#786. So the check appears to have been removed on purpose and this documentation section was left behind describing it.

3. invalid-for-json-schema, docs/errors/usage_errors.md

ImportString has had a JSON schema since 2.8.0 via #9344, so Model.model_json_schema() succeeds and returns a plain string schema. The assert exc_info.code == 'invalid-for-json-schema' inside the except block has not executed since that release.

Output of the example code below

Identical on 2.13.5 and on main at 2.14.0b2:

1. no exception; num = 0j
2. no exception; accepted 'foo'
3. no exception; {'properties': {'a': {'title': 'A', 'type': 'string'}}, 'required': ['a'], 'title': 'ImportStringModel', 'type': 'object'}

Suggested fixes

  1. complex_type is a one-word fix. Model(num='not a complex number') still raises complex_type, and so do [], {} and a bare object(). Replacing False with any of those restores the example.
  2. string_sub_type needs a decision rather than a fix, which is the main reason I am opening an issue instead of sending a patch. Given the trail through #6261, the section probably wants deleting rather than repairing, and possibly the error variant too, but that is your call.
  3. invalid-for-json-schema needs a type that genuinely has no JSON schema, in place of ImportString.

Separately, the blind spot itself is fixable. If an example on these pages had to demonstrate that the error occurred, for instance by printing inside the except and also failing loudly when nothing was raised, then a future change like #12498 would turn the docs red instead of silently making a section untrue. I am happy to send a PR for any or all of this, including that check as a test, if you tell me which direction you prefer.

I used AI assistance to run this audit, in line with your AI policy. I have reproduced and understood each of the three cases myself and can explain any of them.

Example Code

from enum import Enum

from pydantic import (
    BaseModel,
    Field,
    ImportString,
    PydanticUserError,
    ValidationError,
)

# 1. validation_errors.md, `complex_type`: documented to print 'complex_type'
class ComplexModel(BaseModel):
    num: complex

try:
    ComplexModel(num=False)
    print('1. no exception; num =', ComplexModel(num=False).num)
except ValidationError as exc:
    print('1.', repr(exc.errors()[0]['type']))

# 2. validation_errors.md, `string_sub_type`: documented to print 'string_sub_type'
class MyEnum(str, Enum):
    foo = 'foo'

class StrictStrModel(BaseModel):
    x: str = Field(strict=True)

try:
    print('2. no exception; accepted', repr(StrictStrModel(x=MyEnum.foo).x))
except ValidationError as exc:
    print('2.', repr(exc.errors()[0]['type']))

# 3. usage_errors.md, `invalid-for-json-schema`: the assert is never reached
class ImportStringModel(BaseModel):
    a: ImportString

try:
    print('3. no exception;', ImportStringModel.model_json_schema())
except PydanticUserError as exc:
    print('3.', exc.code)

Python, Pydantic & OS Version

pydantic version: 2.13.5
        pydantic-core version: 2.46.5
        python version: 3.12.10 (CPython)
        platform: Windows 11

also reproduced against main at pydantic 2.14.0b2 / pydantic-core 2.49.0

The two patches

Our patch+81 −28
docs/errors/usage_errors.mdindex 7e28e9f8..48403726 100644@@ -620,11 +620,13 @@ type C = B This error is raised when Pydantic fails to generate a JSON schema for some `CoreSchema`.  ```python-from pydantic import BaseModel, ImportString, PydanticUserError+from collections.abc import Callable++from pydantic import BaseModel, PydanticUserError   class Model(BaseModel):-    a: ImportString+    a: Callable   try:docs/errors/validation_errors.mdindex 71d3a929..95e240e7 100644@@ -233,7 +233,7 @@ class Model(BaseModel):   try:-    Model(num=False)+    Model(num='not a complex number') except ValidationError as exc:     print(repr(exc.errors()[0]['type']))     #> 'complex_type'@@ -1845,31 +1845,6 @@ except ValidationError as exc:     #> 'string_pattern_mismatch' ``` -## `string_sub_type`--This error is raised when the value is an instance of a strict subtype of `str` when the field is strict:--```python-from enum import Enum--from pydantic import BaseModel, Field, ValidationError---class MyEnum(str, Enum):-    foo = 'foo'---class Model(BaseModel):-    x: str = Field(strict=True)---try:-    Model(x=MyEnum.foo)-except ValidationError as exc:-    print(repr(exc.errors()[0]['type']))-    #> 'string_sub_type'-```- ## `string_too_long`  This error is raised when the input value is a string whose length is greater than the field's `max_length` constraint:tests/test_docs.pyindex 7c185657..3fa4cd55 100644@@ -5,6 +5,7 @@ import platform import re import subprocess import sys+from collections.abc import Callable from datetime import datetime from pathlib import Path from tempfile import NamedTemporaryFile@@ -13,6 +14,7 @@ from typing import Any import pytest from pydantic_core import core_schema +from pydantic import BaseModel, Field, ImportString, PydanticUserError, ValidationError from pydantic.errors import PydanticErrorCodes  if sys.platform != 'emscripten' and platform.python_implementation() != 'PyPy':@@ -259,12 +261,86 @@ def test_error_codes():     assert code_error_codes == documented_error_codes, 'Error codes in code and docs do not match'  +def _doc_section(md_path, header):+    """Return the text of the markdown section that starts at ``header``."""+    lines = md_path.read_text().splitlines()+    for i, line in enumerate(lines):+        if line.strip() == header:+            break+    else:+        raise AssertionError(f'section {header!r} not found in {md_path.name}')+    for end in range(i + 1, len(lines)):+        if lines[end].startswith('## '):+            break+    else:+        end = len(lines)+    return '\n'.join(lines[i:end])+++def test_error_reference_examples_document_real_errors():+    """Regression for #13802.++    Every error-reference example does something that should fail, catches it, and+    asserts/prints the error code inside the ``except`` branch. When the operation+    stops failing that branch never runs, so the example still exits 0 and the docs+    example suite stays green while documenting behaviour that no longer exists.+    Three examples had silently stopped raising. This test pins the behaviour each+    fixed example relies on and guards the examples from reusing the inputs that had+    quietly stopped raising.+    """+    validation_md = DOCS_ROOT / 'errors/validation_errors.md'+    usage_md = DOCS_ROOT / 'errors/usage_errors.md'++    # 1. `complex_type`: `Model(num=False)` stopped raising -- it now validates to+    #    0j (#12498) -- so the example must use an input that still fails.+    class ComplexTypeExample(BaseModel):+        num: complex++    assert ComplexTypeExample(num=False).num == 0j+    with pytest.raises(ValidationError) as complex_exc:+        ComplexTypeExample(num='not a complex number')+    assert complex_exc.value.errors()[0]['type'] == 'complex_type'++    complex_section = _doc_section(validation_md, '## `complex_type`')+    assert 'num=False' not in complex_section+    assert "#> 'complex_type'" in complex_section++    # 2. `string_sub_type`: no str schema rejects a str subclass anymore+    #    (see #6261 / pydantic-core#786), so the section was removed entirely.+    class StrSubclass(str):+        pass++    class StrictStrExample(BaseModel):+        x: str = Field(strict=True)++    assert StrictStrExample(x=StrSubclass('foo')).x == 'foo'+    assert '## `string_sub_type`' not in validation_md.read_text()++    # 3. `invalid-for-json-schema`: `ImportString` gained a JSON schema (#9344), so+    #    the example needs a type that genuinely has none.+    class ImportStringExample(BaseModel):+        a: ImportString++    assert ImportStringExample.model_json_schema()['properties']['a']['type'] == 'string'++    class NoJsonSchemaExample(BaseModel):+        a: Callable++    with pytest.raises(PydanticUserError) as usage_exc:+        NoJsonSchemaExample.model_json_schema()+    assert usage_exc.value.code == 'invalid-for-json-schema'++    usage_section = _doc_section(usage_md, '## JSON schema invalid type {#invalid-for-json-schema}')+    assert 'ImportString' not in usage_section++ def test_validation_error_codes():     error_text = (DOCS_ROOT / 'errors/validation_errors.md').read_text()      expected_validation_error_codes = set(core_schema.ErrorType.__args__)     # Remove codes that are not currently accessible from pydantic:     expected_validation_error_codes.remove('timezone_offset')  # not currently exposed for configuration in pydantic+    expected_validation_error_codes.remove('string_sub_type')  # no str schema raises this anymore; see #13802 / pydantic-core#786      test_failures = []  
Merged fix · PR #+6 −33
docs/concepts/experimental.mdindex cc349f3a0..ab1665260 100644@@ -342,7 +342,7 @@ class MyModel(BaseModel): ta = TypeAdapter(MyModel) try:     v = ta.validate_json(-        '{"a": 1, "b": ["12345", "12', experimental_allow_partial=True+        '{"a": 1, "b": ["12345", "12"', experimental_allow_partial=True     ) except ValidationError as e:     print(e)docs/errors/usage_errors.mdindex 7e28e9f82..3e6a67412 100644@@ -620,11 +620,13 @@ type C = B This error is raised when Pydantic fails to generate a JSON schema for some `CoreSchema`.  ```python-from pydantic import BaseModel, ImportString, PydanticUserError+from collections.abc import Callable++from pydantic import BaseModel, PydanticUserError   class Model(BaseModel):-    a: ImportString+    a: Callable[..., object]   try:docs/errors/validation_errors.mdindex 71d3a9292..95e240e70 100644@@ -233,7 +233,7 @@ class Model(BaseModel):   try:-    Model(num=False)+    Model(num='not a complex number') except ValidationError as exc:     print(repr(exc.errors()[0]['type']))     #> 'complex_type'@@ -1845,31 +1845,6 @@ except ValidationError as exc:     #> 'string_pattern_mismatch' ``` -## `string_sub_type`--This error is raised when the value is an instance of a strict subtype of `str` when the field is strict:--```python-from enum import Enum--from pydantic import BaseModel, Field, ValidationError---class MyEnum(str, Enum):-    foo = 'foo'---class Model(BaseModel):-    x: str = Field(strict=True)---try:-    Model(x=MyEnum.foo)-except ValidationError as exc:-    print(repr(exc.errors()[0]['type']))-    #> 'string_sub_type'-```- ## `string_too_long`  This error is raised when the input value is a string whose length is greater than the field's `max_length` constraint:pydantic-core/python/pydantic_core/core_schema.pyindex b6fce060b..fda0e35ee 100644@@ -4776,7 +4776,6 @@ ErrorType: TypeAlias = Literal[     'iterable_type',     'iteration_error',     'string_type',-    'string_sub_type',     'string_unicode',     'string_too_short',     'string_too_long',pydantic-core/src/errors/types.rsindex 4bdb92135..f49f48509 100644@@ -247,7 +247,6 @@ error_types! {     // ---------------------     // string errors     StringType {},-    StringSubType {},     StringUnicode {},     StringTooShort {         min_length: {ctx_type: usize, ctx_fn: field_from_context},@@ -533,7 +532,6 @@ impl ErrorType {             Self::IterableType { .. } => "Input should be iterable",             Self::IterationError { .. } => "Error iterating over object, error: {error}",             Self::StringType { .. } => "Input should be a valid string",-            Self::StringSubType { .. } => "Input should be a string, not an instance of a subclass of str",             Self::StringUnicode { .. } => {                 "Input should be a valid string, unable to parse raw data as a unicode string"             }pydantic-core/tests/test_errors.pyindex 66f15d81a..b024e618b 100644@@ -303,7 +303,6 @@ all_errors = [         {'field_type': 'Foobar', 'max_length': 42, 'actual_length': 50},     ),     ('string_type', 'Input should be a valid string', None),-    ('string_sub_type', 'Input should be a string, not an instance of a subclass of str', None),     ('string_unicode', 'Input should be a valid string, unable to parse raw data as a unicode string', None),     ('string_pattern_mismatch', "String should match pattern 'foo'", {'pattern': 'foo'}),     ('string_not_ascii', 'String should contain only ASCII characters', None), 

What the agent reported

Review passed with no correctness issues. The work is complete.

Summary

What was wrong

Three examples on Pydantic's error-reference pages documented errors that the code no longer raises, and the docs test suite couldn't detect it (each example only asserts/prints inside its except branch, so when nothing raises, the example silently exits 0):

  1. complex_type — Model(num=False) now validates to 0j (changed in #12498).
  2. string_sub_type — no str schema rejects a str subclass anymore (see #6261 / pydantic-core#786).
  3. invalid-for-json-schema — ImportString gained a JSON schema (#9344), so model_json_schema() succeeds.

I verified all three empirically in the Linux container (pydantic 2.14.0b2 / pydantic-core 2.49.0) before changing anything.

What I changed

  • docs/errors/validation_errors.md: complex_type input Model(num=False) → Model(num='not a complex number') (still prints #> 'complex_type'); deleted the entire string_sub_type section (it cannot be made truthful — no str schema raises it).
  • docs/errors/usage_errors.md: invalid-for-json-schema example now uses collections.abc.Callable (genuinely has no JSON schema) instead of ImportString.
  • tests/test_docs.py: added expected_validation_error_codes.remove('string_sub_type') in test_validation_error_codes — required because string_sub_type is still in core_schema.ErrorType and that test demands documented codes exactly match ErrorType; this mirrors the existing sanctioned timezone_offset removal. Added a robust regression test test_error_reference_examples_document_real_errors.

Why the regression test isn't the exec-based "blind-spot" check the report proposed: I built and tested that version. It fails unreliably in the full suite because hundreds of prior docs examples exec class Model and pollute pydantic's process-global schema state, corrupting my exec'd model (pytest_examples avoids this via per-example module isolation). The shipped test instead reproduces each scenario with unique class names in normal module scope and guards the docs against reusing the broken inputs — order-independent and reliable.

Test evidence

  • ./bt-test tests/test_docs.py: 621 passed, 108 skipped, 0 failed — includes the new regression test, test_validation_error_codes, test_error_codes, and the actual execution (via pytest_examples) of the fixed complex_type and invalid-for-json-schema examples.
  • Fail-before is guaranteed by the guards ('num=False' not in, '## \string_sub_type\' not in, 'ImportString' not in), all of which held in the original docs (confirmed by reading them).
  • Along the way I caught and fixed a ruff UP035 failure (typing.Callable → collections.abc.Callable). All probe/scratch files removed; git status shows only the three intended files.

Requirements disposition

  1. Fix complex_type — ✅ Implemented ('not a complex number'); verified raises complex_type; example runs green under pytest_examples.
  2. Decide + handle string_sub_type — ✅ Deleted the section (evidence: verified no str schema — strict field, config-strict, raw core_schema strict, strict+pattern — rejects a str subclass); coordinated test_validation_error_codes update so the code/docs match stays truthful.
  3. Fix invalid-for-json-schema — ✅ Implemented (Callable); verified raises code='invalid-for-json-schema'; example runs green.
  4. Add a test so future silent regressions turn docs red (optional) — ✅ Addressed via test_error_reference_examples_document_real_errors, which pins each behavior and guards the examples. Noted deviation: not the generic exec-over-all-examples check the report floated, because that proved unreliable under the suite's process-global schema pollution.