Backtestspydanticrun-20260929-204949#13419

Issue #13419 Merged fix PR #13438 Run 29 Sep 2026 · run-20260929-204949 pydantic/pydantic

"dictionary changed size during iteration" during model_rebuild

Worse

The merged fix was judged better than our patch.

In this judgment, Patch A = our agent, Patch B = the merged fix.

The issue is a race: concurrent `model_rebuild()` calls mutate the class `__dict__` (delattr/setattr of schema/validator/serializer) while another thread iterates `vars(cls)` during annotation evaluation. Patch A only patches the single visible symptom — it makes `LazyLocalNamespace.data` take an atomic copy so iteration won't raise `RuntimeError: dictionary changed size`. But the issue text explicitly notes the same race also produces `AttributeError: ... has no attribute '__pydantic_core_schema__'` from the delattr block, plus the risk of a parent validator being resolved mid-mutation; Patch A does nothing about those. It masks one manifestation of an unsynchronized mutation. Patch B attacks the root cause by serializing rebuilds with a global reentrant lock, with double-checked completeness (unlocked fast path, re-check inside the lock), and correctly reasons about why the lock must be reentrant (nested rebuild via `__class_getitem__` parametrization) and global rather than per-class (deadlock avoidance). It also broadens the mock check to `(MockCoreSchema, MockValSer)` so mock attrs aren't deleted, preventing concurrent readers from resolving a parent's validator. With rebuilds serialized, no thread iterates `vars(cls)` while another mutates it, so the dictionary-changed-size crash, the AttributeError, and the wrong-validator races are all eliminated. Tests: Patch A's test simulates a mutation-during-iteration mapping — reasonable but narrow. Patch B adds a reentrancy test (guards against the lock deadlocking on nested parametrization rebuilds) and a genuine two-thread serialization test that gates schema generation to prove the second thread blocks on the lock and returns `None` after the first completes, exercising the double-checked path and the on-complete hook firing once. Patch B's tests better exercise the actual concurrency semantics being fixed. Scope: Patch B touches more (main.py, dataclasses.py) but each change is on-point to the race. Patch B is the stronger, root-cause fix.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

6m 37s10 turns

Wall-clock time in the container

Full test suite

No new failures

12,260 passed · 501 skipped · 12,761 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

30vs 168 merged

Changed lines, ours vs merged

Initial Checks

  • I confirm that I'm using Pydantic V2

Description

This just hit us in production:

Traceback (most recent call last):
  ...
  File "fastapi/routing.py", line 120, in app
    response = await f(request)
  File "fastapi/routing.py", line 674, in app
    raw_response = await run_endpoint_function(
  File "fastapi/routing.py", line 330, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "starlette/concurrency.py", line 34, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
  File "anyio/to_thread.py", line 63, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "anyio/_backends/_asyncio.py", line 2518, in run_sync_in_worker_thread
    return await future
  File "anyio/_backends/_asyncio.py", line 1002, in run
    result = context.run(func, *args)
  ...
  File "...", line 157, in ...
    MyPydanticClass(
  File "pydantic/_internal/_mock_val_ser.py", line 165, in handler
    if cls.model_rebuild(raise_errors=False, _parent_namespace_depth=5) is not False:
  File "pydantic/main.py", line 683, in model_rebuild
    return _model_construction.complete_model_class(
  File "pydantic/_internal/_model_construction.py", line 642, in complete_model_class
    cls.__pydantic_fields__, cls.__pydantic_extra_info__ = rebuild_model_fields(
  File "pydantic/_internal/_fields.py", line 464, in rebuild_model_fields
    new_field = _recreate_field_info(
  File "pydantic/_internal/_fields.py", line 502, in _recreate_field_info
    ann = _typing_extra.eval_type(
  File "pydantic/_internal/_typing_extra.py", line 442, in eval_type
    return eval_type_backport(value, globalns, localns)
  File "pydantic/_internal/_typing_extra.py", line 474, in eval_type_backport
    return _eval_type_backport(value, globalns, localns, type_params)
  File "pydantic/_internal/_typing_extra.py", line 511, in _eval_type_backport
    return _eval_type(value, globalns, localns, type_params)
  File "pydantic/_internal/_typing_extra.py", line 543, in _eval_type
    evaluated = typing._eval_type(  # type: ignore
  File "typing.py", line 461, in _eval_type
    return evaluate_forward_ref(t, globals=globalns, locals=localns,
  File "typing.py", line 1000, in evaluate_forward_ref
    value = forward_ref.evaluate(globals=globals, locals=locals,
  File "annotationlib.py", line 207, in evaluate
    return eval(code, globals=globals, locals=locals)
  File "<string>", line 1, in <module>
    from __future__ import annotations
  File "pydantic/_internal/_namespace_utils.py", line 97, in __getitem__
    return self.data[key]
  File "functools.py", line 1126, in __get__
    val = self.func(instance)
  File "pydantic/_internal/_namespace_utils.py", line 91, in data
    return {k: v for ns in self._namespaces for k, v in ns.items()}
RuntimeError: dictionary changed size during iteration

Reproducer below.

Example Code

"""
NOTE: Repro script generated with Claude Fable

Repro: concurrent lazy `model_rebuild()` raises
`RuntimeError: dictionary changed size during iteration`.

Setup: a module using `from __future__ import annotations` defines models whose
field types are only defined *later* in the same module. Building such a model's
schema fails during class creation, so pydantic leaves it incomplete (mock
validator) and defers the build to the first instantiation via
`model_rebuild(raise_errors=False)`.

If the first instantiations happen concurrently (e.g. FastAPI's threadpool
serving the first requests after process start), multiple threads enter
`model_rebuild()` for the same class simultaneously. `model_rebuild()` mutates
the class `__dict__` (delattr/setattr of `__pydantic_core_schema__`,
`__pydantic_validator__`, `__pydantic_serializer__`), while
`NsResolver.types_namespace` includes the live `vars(cls)` of that class in the
`LazyLocalNamespace` another thread is iterating during annotation evaluation
(`LazyLocalNamespace.data`). Result:

    RuntimeError: dictionary changed size during iteration

The same race also intermittently produces
    AttributeError: type object 'Box0' has no attribute '__pydantic_core_schema__'
from the delattr block at the top of `model_rebuild()`.

Each round executes the module source into a fresh module (fresh, incomplete
classes -- like a freshly started process) and races threads instantiating the
models. Typically fires within a few dozen rounds (< 1 min).

Run: python pydantic_rebuild_race_repro.py
"""

import sys
import threading
import time
import traceback
import types

import pydantic

print(f"python {sys.version}")
print(f"pydantic {pydantic.VERSION}")

sys.setswitchinterval(1e-5)  # more frequent thread preemption; widens the race window

N_GENERICS = 2  # generic origins shared by all models (parametrizing triggers origin.model_rebuild)
N_LEAVES = 16  # leaf models, defined at the end of the module so everything before stays incomplete
N_MODELS = 4  # top-level models, the ones instantiated by the racing threads
N_METHODS = 60  # dummy methods to enlarge each class __dict__ (the dict iterated during annotation eval)
N_THREADS = 32
ROUNDS = 200
RACE_WINDOW_S = 0.5


def make_module_source() -> str:
    boxes = " | ".join(f"Box{g}[Leaf{l}]" for g in range(N_GENERICS) for l in range(N_LEAVES))
    methods = [f"    def method{m}(self) -> None: ..." for m in range(N_METHODS)]
    lines = ["from __future__ import annotations", "from pydantic import BaseModel", ""]
    # Top-level models come first: all their field types are still undefined here,
    # so their schema build is deferred until first instantiation.
    for i in range(N_MODELS):
        lines += [f"class Model{i}(BaseModel):", f"    items: list[{boxes}] = []", *methods, ""]
    for g in range(N_GENERICS):
        lines += [f"class Box{g}[T](BaseModel):", "    value: T | None = None", "    ref: Leaf0 | None = None", *methods, ""]
    for l in range(N_LEAVES):
        lines += [f"class Leaf{l}(BaseModel):", "    x: int = 0", ""]
    return "\n".join(lines)


MODULE_SOURCE = make_module_source()

errors: list[str] = []
errors_lock = threading.Lock()


def worker(barrier: threading.Barrier, model: type) -> None:
    barrier.wait()
    end = time.monotonic() + RACE_WINDOW_S
    while time.monotonic() < end:
        try:
            model()  # first call hits the mock validator -> lazy model_rebuild()
        except Exception:
            with errors_lock:
                errors.append(traceback.format_exc())


for rnd in range(ROUNDS):
    # Fresh module => fresh incomplete classes, like a freshly started process.
    mod = types.ModuleType(f"repro_generated_{rnd}")
    sys.modules[mod.__name__] = mod
    exec(compile(MODULE_SOURCE, mod.__name__, "exec"), mod.__dict__)
    models = [getattr(mod, f"Model{i}") for i in range(N_MODELS)]
    assert not models[0].__pydantic_complete__

    barrier = threading.Barrier(N_THREADS)
    threads = [threading.Thread(target=worker, args=(barrier, models[i % N_MODELS])) for i in range(N_THREADS)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    hits = [tb for tb in errors if "changed size during iteration" in tb]
    if hits:
        print(hits[0], file=sys.stderr)
        print(f"REPRODUCED in round {rnd}: RuntimeError: dictionary changed size during iteration")
        sys.exit(1)
    if errors:
        first_line = errors[0].strip().splitlines()[-1]
        print(f"round {rnd}: {len(errors)} sibling race errors (e.g. {first_line})")
        errors.clear()

print("not reproduced")

Python, Pydantic & OS Version

In Prod: Ubuntu Docker, in Dev:

             pydantic version: 2.13.4
        pydantic-core version: 2.46.4
          pydantic-core build: profile=release pgo=false
               python version: 3.14.6 | packaged by conda-forge | (main, Jun 12 2026, 08:47:56) [Clang 20.1.8 ]
                     platform: macOS-26.5.1-arm64-arm-64bit-Mach-O
             related packages: pyright-1.1.410 fastapi-0.136.3 pydantic-extra-types-2.11.1 email-validator-2.3.0 pydantic-settings-2.14.1 typing_extensions-4.15.0
                       commit: unknown

The two patches

Our patch+29 −1
pydantic/_internal/_namespace_utils.pyindex d308f7e4..691f7ea1 100644@@ -88,7 +88,17 @@ class LazyLocalNamespace(Mapping[str, Any]):      @cached_property     def data(self) -> dict[str, Any]:-        return {k: v for ns in self._namespaces for k, v in ns.items()}+        data: dict[str, Any] = {}+        for ns in self._namespaces:+            # A namespace can be a live mapping (e.g. a class `__dict__`) that+            # another thread mutates while we read it -- this happens when a+            # `model_rebuild()` call runs concurrently for the class being+            # inspected. Iterating such a namespace directly can raise+            # `RuntimeError: dictionary changed size during iteration`, so we+            # take an atomic copy first (`dict`/`mappingproxy` copies don't call+            # back into Python and can't observe a partially-mutated state).+            data.update(ns.copy() if hasattr(ns, 'copy') else ns)+        return data      def __len__(self) -> int:         return len(self.data)tests/test_forward_ref.pyindex 9b0325d2..2875b12b 100644@@ -1144,6 +1144,24 @@ def test_lazy_local_namespace_len() -> None:     assert len(namespace) == 2  +def test_lazy_local_namespace_data_mutated_during_iteration() -> None:+    # A namespace passed to `LazyLocalNamespace` can be a live mapping (e.g. a+    # class `__dict__`) that another thread mutates while `.data` reads it, as+    # happens when a `model_rebuild()` runs concurrently for the class being+    # inspected. `.data` must not raise `RuntimeError: dictionary changed size+    # during iteration` (concurrent model_rebuild).+    class MutatingNamespace(dict):+        def items(self):  # type: ignore[override]+            for key, value in super().items():+                self[object()] = None  # simulate a concurrent mutation+                yield key, value++    namespace = LazyLocalNamespace(MutatingNamespace({'a': int, 'b': str}))+    # Reading a key builds `.data`; this must succeed instead of raising.+    assert namespace['a'] is int+    assert namespace['b'] is str++ @pytest.mark.xfail(     reason='While `get_cls_type_hints` uses the correct module ns for each base, `collect_model_fields` '     'will still use the `FieldInfo` instances from each base (see the `parent_fields_lookup` logic). ' 
Merged fix · PR #13438+138 −30
pydantic/dataclasses.pyindex 62d380f54..474289ec3 100644@@ -330,9 +330,13 @@ def rebuild_dataclass(         return None      for attr in ('__pydantic_core_schema__', '__pydantic_validator__', '__pydantic_serializer__'):-        if attr in cls.__dict__ and not isinstance(getattr(cls, attr), _mock_val_ser.MockValSer):+        if attr in cls.__dict__ and not isinstance(+            getattr(cls, attr), (_mock_val_ser.MockCoreSchema, _mock_val_ser.MockValSer)+        ):             # Deleting the validator/serializer is necessary as otherwise they can get reused in             # pydantic-core. Same applies for the core schema that can be reused in schema generation.+            # We do so only if they aren't mock instances, otherwise concurrent reads of these attributes+            # (e.g. when instantiating the dataclass in another thread) can resolve them from the parent class.             delattr(cls, attr)      cls.__pydantic_complete__ = Falsepydantic/main.pyindex 758a839c6..6bb1554e5 100644@@ -8,6 +8,7 @@ from __future__ import annotations as _annotations  import operator import sys+import threading import types import warnings from collections.abc import Generator, Mapping@@ -78,6 +79,18 @@ IncEx: TypeAlias = set[int] | set[str] | Mapping[int, Union['IncEx', bool]] | Ma  _object_setattr = _model_construction.object_setattr +_rebuild_lock = threading.RLock()+"""+A lock used to make model rebuilds thread-safe when first instantiating an incomplete model.++Rebuilding a model isn't thread-safe (the class attributes are mutated during the rebuild,+while other threads may be reading them to perform validation/serialization), so `model_rebuild()`+calls are serialized using this lock. The lock is reentrant, as rebuilding a model can trigger the+rebuild of another one (e.g. when a generic origin is rebuilt during parametrization in+`__class_getitem__()`). For the same reason, the lock is global and not per-class: two threads+holding their own class's lock could otherwise request the other's and deadlock.+"""+  def _check_frozen(model_cls: type[BaseModel], name: str, value: Any) -> None:     if model_cls.model_config.get('frozen'):@@ -681,42 +694,54 @@ class BaseModel(metaclass=_model_construction.ModelMetaclass):             Returns `None` if the schema is already "complete" and rebuilding was not required.             If rebuilding _was_ required, returns `True` if rebuilding was successful, otherwise `False`.         """-        already_complete = cls.__pydantic_complete__-        if already_complete and not force:+        # As the rebuild lock is global (not per model class), avoid needlessly holding it if the model is+        # already complete:+        if cls.__pydantic_complete__ and not force:             return None -        cls.__pydantic_complete__ = False+        with _rebuild_lock:+            # Re-check inside the lock, as another thread may have rebuilt the model while we were waiting:+            already_complete = cls.__pydantic_complete__+            if already_complete and not force:+                return None -        for attr in ('__pydantic_core_schema__', '__pydantic_validator__', '__pydantic_serializer__'):-            if attr in cls.__dict__ and not isinstance(getattr(cls, attr), _mock_val_ser.MockValSer):-                # Deleting the validator/serializer is necessary as otherwise they can get reused in-                # pydantic-core. We do so only if they aren't mock instances, otherwise — as `model_rebuild()`-                # isn't thread-safe — concurrent model instantiations can lead to the parent validator being used.-                # Same applies for the core schema that can be reused in schema generation.-                delattr(cls, attr)+            cls.__pydantic_complete__ = False -        if _types_namespace is not None:-            rebuild_ns = _types_namespace-        elif _parent_namespace_depth > 0:-            rebuild_ns = _typing_extra.parent_frame_namespace(parent_depth=_parent_namespace_depth, force=True) or {}-        else:-            rebuild_ns = {}+            for attr in ('__pydantic_core_schema__', '__pydantic_validator__', '__pydantic_serializer__'):+                if attr in cls.__dict__ and not isinstance(+                    getattr(cls, attr), (_mock_val_ser.MockCoreSchema, _mock_val_ser.MockValSer)+                ):+                    # Deleting the validator/serializer is necessary as otherwise they can get reused in+                    # pydantic-core. Same applies for the core schema that can be reused in schema generation.+                    # We do so only if they aren't mock instances, otherwise concurrent reads of these attributes+                    # — performed without holding the rebuild lock (e.g. when instantiating the model) — can+                    # resolve them from the parent class.+                    delattr(cls, attr)++            if _types_namespace is not None:+                rebuild_ns = _types_namespace+            elif _parent_namespace_depth > 0:+                rebuild_ns = (+                    _typing_extra.parent_frame_namespace(parent_depth=_parent_namespace_depth, force=True) or {}+                )+            else:+                rebuild_ns = {} -        parent_ns = _model_construction.unpack_lenient_weakvaluedict(cls.__pydantic_parent_namespace__) or {}+            parent_ns = _model_construction.unpack_lenient_weakvaluedict(cls.__pydantic_parent_namespace__) or {} -        ns_resolver = _namespace_utils.NsResolver(-            parent_namespace={**rebuild_ns, **parent_ns},-        )+            ns_resolver = _namespace_utils.NsResolver(+                parent_namespace={**rebuild_ns, **parent_ns},+            ) -        return _model_construction.complete_model_class(-            cls,-            _config.ConfigWrapper(cls.model_config, check=False),-            ns_resolver,-            raise_errors=raise_errors,-            # If the model was already complete, we don't need to call the hook again.-            call_on_complete_hook=not already_complete,-            is_force_rebuild=force,-        )+            return _model_construction.complete_model_class(+                cls,+                _config.ConfigWrapper(cls.model_config, check=False),+                ns_resolver,+                raise_errors=raise_errors,+                # If the model was already complete, we don't need to call the hook again.+                call_on_complete_hook=not already_complete,+                is_force_rebuild=force,+            )      @classmethod     def model_validate(tests/test_main.pyindex c87dae793..f924cec99 100644@@ -2,6 +2,7 @@ import json import platform import re import sys+import threading import typing import warnings from collections import defaultdict@@ -2541,6 +2542,84 @@ def test_model_rebuild_localns():         C.model_rebuild(_types_namespace={'A': A})  +@pytest.mark.timeout(5)+def test_model_rebuild_nested_during_parametrization():+    T = TypeVar('T')++    class Model(BaseModel):+        b: Optional['Box[Leaf]'] = None++    class Box(BaseModel, Generic[T]):+        leaf: Optional['Leaf'] = None++    class Leaf(BaseModel):+        x: int = 0++    assert not Model.__pydantic_complete__+    assert not Box.__pydantic_complete__++    # Rebuilding `Model` evaluates `Box[Leaf]`, and `__class_getitem__` then calls+    # `Box.model_rebuild()` on the same thread. Any locking added to the rebuild logic+    # must support this reentrancy:+    m = Model(b={'leaf': {'x': 1}})+    assert m.b.leaf.x == 1+    assert Model.__pydantic_complete__+++@pytest.mark.skipif(sys.platform == 'emscripten', reason='no threading on emscripten')+def test_model_rebuild_already_rebuilt_by_other_thread():+    schema_gen_entered = threading.Event()+    resume_schema_gen = threading.Event()+    hook_calls: list[type] = []++    class Model(BaseModel):+        x: 'Gate'++        @classmethod+        def __pydantic_on_complete__(cls) -> None:+            hook_calls.append(cls)++    class Gate:+        @classmethod+        def __get_pydantic_core_schema__(cls, source_type, handler):+            schema_gen_entered.set()+            resume_schema_gen.wait(timeout=10)+            return core_schema.int_schema()++    assert not Model.__pydantic_complete__++    results: dict[str, bool | None] = {}+    t2_started = threading.Event()++    def rebuild(key: str) -> None:+        if key == 't2':+            t2_started.set()+        results[key] = Model.model_rebuild(_types_namespace={'Gate': Gate})++    # The first thread acquires the rebuild lock and waits in schema generation:+    t1 = threading.Thread(target=rebuild, args=('t1',), daemon=True)+    t1.start()+    assert schema_gen_entered.wait(timeout=10)++    # While the first thread is waiting on `resume_schema_gen`, `Model` can't become complete,+    # so the second thread is guaranteed to pass the unlocked completeness check and block on the lock:+    t2 = threading.Thread(target=rebuild, args=('t2',), daemon=True)+    t2.start()+    assert t2_started.wait(timeout=10)+    assert t2.is_alive()+    assert 't2' not in results++    resume_schema_gen.set()+    t1.join(timeout=10)+    t2.join(timeout=10)+    assert not t1.is_alive()+    assert not t2.is_alive()++    assert results == {'t1': True, 't2': None}+    assert hook_calls == [Model]+    assert Model(x=1).x == 1++ def test_model_rebuild_zero_depth():     class Model(BaseModel):         x: 'X_Type' 

What the agent reported

The work is complete. Here is my final report.

What was wrong

Concurrent lazy model_rebuild() calls raised RuntimeError: dictionary changed size during iteration in LazyLocalNamespace.data (pydantic/_internal/_namespace_utils.py).

NsResolver.types_namespace appends the live class __dict__ (vars(typ), a mappingproxy) into a LazyLocalNamespace (line 276). LazyLocalNamespace.data then iterated that mapping live via a dict comprehension over ns.items(). When a second thread ran model_rebuild() for the same class — deleting/setting __pydantic_core_schema__, __pydantic_validator__, __pydantic_serializer__ on the class __dict__ — the in-flight iteration observed the size change and crashed. FastAPI's threadpool serving the first requests after process start is a natural trigger.

What I changed

pydantic/_internal/_namespace_utils.py — rewrote LazyLocalNamespace.data to copy each namespace before merging instead of iterating it live:

data: dict[str, Any] = {}
for ns in self._namespaces:
    data.update(ns.copy() if hasattr(ns, 'copy') else ns)
return data

dict.copy() / mappingproxy.copy() are atomic C-level operations that don't call back into Python and can't observe a partially-mutated state (I verified mappingproxy.copy() delegates to the underlying dict.copy(), and a dict subclass's .copy() returns a plain dict without invoking any overridden items()). Merge semantics are unchanged — later namespaces still override earlier ones.

tests/test_forward_ref.py — added test_lazy_local_namespace_data_mutated_during_iteration, which deterministically (no threads) reproduces the failure using a namespace whose items() mutates itself mid-iteration, then asserts .data still resolves keys.

Test evidence

  • New test fails before the fix with the exact RuntimeError: dictionary changed size during iteration; passes after.
  • ./bt-test tests/test_forward_ref.py::test_lazy_local_namespace_data_mutated_during_iteration tests/test_forward_ref.py::test_lazy_local_namespace_len → 2 passed.
  • Full tests/test_forward_ref.py → 68 passed, 4 pre-existing xfailed.
  • Blast radius tests/test_internal.py tests/test_construction.py → 81 passed, 2 pre-existing xfailed.
  • Reviewer confirmed the fix resolves the reported bug, preserves merge/len semantics, changes nothing unrelated, and the test genuinely distinguishes old vs. new behavior.

Requirements disposition

  • Fix RuntimeError: dictionary changed size during iteration during concurrent model_rebuild — ✅ Implemented and covered by the new regression test (fails before, passes after).
  • Reproducer script provided in report — ✅ Its failure mode is captured deterministically by the new unit test. I did not run the flaky, timing-dependent 200-round threaded script as a gate (it targets 3.14 and can take up to a minute per run); the deterministic test exercises the identical code path and mutation-during-iteration condition.
  • The related intermittent AttributeError: ... has no attribute '__pydantic_core_schema__' (from the delattr block in model_rebuild) — mentioned in the report only as a secondary symptom of the same race, not requested to be fixed. It originates in a different location (main.py/_model_construction.py, not _namespace_utils.py) and is out of scope for this LazyLocalNamespace.data fix; I did not change it.

All changes are left uncommitted in the working tree.