Backtestspydanticrun-20260929-204949#13419
"dictionary changed size during iteration" during model_rebuild
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The issue is a race: concurrent `model_rebuild()` calls mutate the class `__dict__` (delattr/setattr of schema/validator/serializer) while another thread iterates `vars(cls)` during annotation evaluation. Patch A only patches the single visible symptom — it makes `LazyLocalNamespace.data` take an atomic copy so iteration won't raise `RuntimeError: dictionary changed size`. But the issue text explicitly notes the same race also produces `AttributeError: ... has no attribute '__pydantic_core_schema__'` from the delattr block, plus the risk of a parent validator being resolved mid-mutation; Patch A does nothing about those. It masks one manifestation of an unsynchronized mutation. Patch B attacks the root cause by serializing rebuilds with a global reentrant lock, with double-checked completeness (unlocked fast path, re-check inside the lock), and correctly reasons about why the lock must be reentrant (nested rebuild via `__class_getitem__` parametrization) and global rather than per-class (deadlock avoidance). It also broadens the mock check to `(MockCoreSchema, MockValSer)` so mock attrs aren't deleted, preventing concurrent readers from resolving a parent's validator. With rebuilds serialized, no thread iterates `vars(cls)` while another mutates it, so the dictionary-changed-size crash, the AttributeError, and the wrong-validator races are all eliminated. Tests: Patch A's test simulates a mutation-during-iteration mapping — reasonable but narrow. Patch B adds a reentrancy test (guards against the lock deadlocking on nested parametrization rebuilds) and a genuine two-thread serialization test that gates schema generation to prove the second thread blocks on the lock and returns `None` after the first completes, exercising the double-checked path and the on-complete hook firing once. Patch B's tests better exercise the actual concurrency semantics being fixed. Scope: Patch B touches more (main.py, dataclasses.py) but each change is on-point to the race. Patch B is the stronger, root-cause fix.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
6m 37s10 turns
Wall-clock time in the container
Full test suite
No new failures
12,260 passed · 501 skipped · 12,761 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
30vs 168 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I confirm that I'm using Pydantic V2
Description
This just hit us in production:
Traceback (most recent call last):
...
File "fastapi/routing.py", line 120, in app
response = await f(request)
File "fastapi/routing.py", line 674, in app
raw_response = await run_endpoint_function(
File "fastapi/routing.py", line 330, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
File "starlette/concurrency.py", line 34, in run_in_threadpool
return await anyio.to_thread.run_sync(func)
File "anyio/to_thread.py", line 63, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "anyio/_backends/_asyncio.py", line 2518, in run_sync_in_worker_thread
return await future
File "anyio/_backends/_asyncio.py", line 1002, in run
result = context.run(func, *args)
...
File "...", line 157, in ...
MyPydanticClass(
File "pydantic/_internal/_mock_val_ser.py", line 165, in handler
if cls.model_rebuild(raise_errors=False, _parent_namespace_depth=5) is not False:
File "pydantic/main.py", line 683, in model_rebuild
return _model_construction.complete_model_class(
File "pydantic/_internal/_model_construction.py", line 642, in complete_model_class
cls.__pydantic_fields__, cls.__pydantic_extra_info__ = rebuild_model_fields(
File "pydantic/_internal/_fields.py", line 464, in rebuild_model_fields
new_field = _recreate_field_info(
File "pydantic/_internal/_fields.py", line 502, in _recreate_field_info
ann = _typing_extra.eval_type(
File "pydantic/_internal/_typing_extra.py", line 442, in eval_type
return eval_type_backport(value, globalns, localns)
File "pydantic/_internal/_typing_extra.py", line 474, in eval_type_backport
return _eval_type_backport(value, globalns, localns, type_params)
File "pydantic/_internal/_typing_extra.py", line 511, in _eval_type_backport
return _eval_type(value, globalns, localns, type_params)
File "pydantic/_internal/_typing_extra.py", line 543, in _eval_type
evaluated = typing._eval_type( # type: ignore
File "typing.py", line 461, in _eval_type
return evaluate_forward_ref(t, globals=globalns, locals=localns,
File "typing.py", line 1000, in evaluate_forward_ref
value = forward_ref.evaluate(globals=globals, locals=locals,
File "annotationlib.py", line 207, in evaluate
return eval(code, globals=globals, locals=locals)
File "<string>", line 1, in <module>
from __future__ import annotations
File "pydantic/_internal/_namespace_utils.py", line 97, in __getitem__
return self.data[key]
File "functools.py", line 1126, in __get__
val = self.func(instance)
File "pydantic/_internal/_namespace_utils.py", line 91, in data
return {k: v for ns in self._namespaces for k, v in ns.items()}
RuntimeError: dictionary changed size during iteration
Reproducer below.
Example Code
"""
NOTE: Repro script generated with Claude Fable
Repro: concurrent lazy `model_rebuild()` raises
`RuntimeError: dictionary changed size during iteration`.
Setup: a module using `from __future__ import annotations` defines models whose
field types are only defined *later* in the same module. Building such a model's
schema fails during class creation, so pydantic leaves it incomplete (mock
validator) and defers the build to the first instantiation via
`model_rebuild(raise_errors=False)`.
If the first instantiations happen concurrently (e.g. FastAPI's threadpool
serving the first requests after process start), multiple threads enter
`model_rebuild()` for the same class simultaneously. `model_rebuild()` mutates
the class `__dict__` (delattr/setattr of `__pydantic_core_schema__`,
`__pydantic_validator__`, `__pydantic_serializer__`), while
`NsResolver.types_namespace` includes the live `vars(cls)` of that class in the
`LazyLocalNamespace` another thread is iterating during annotation evaluation
(`LazyLocalNamespace.data`). Result:
RuntimeError: dictionary changed size during iteration
The same race also intermittently produces
AttributeError: type object 'Box0' has no attribute '__pydantic_core_schema__'
from the delattr block at the top of `model_rebuild()`.
Each round executes the module source into a fresh module (fresh, incomplete
classes -- like a freshly started process) and races threads instantiating the
models. Typically fires within a few dozen rounds (< 1 min).
Run: python pydantic_rebuild_race_repro.py
"""
import sys
import threading
import time
import traceback
import types
import pydantic
print(f"python {sys.version}")
print(f"pydantic {pydantic.VERSION}")
sys.setswitchinterval(1e-5) # more frequent thread preemption; widens the race window
N_GENERICS = 2 # generic origins shared by all models (parametrizing triggers origin.model_rebuild)
N_LEAVES = 16 # leaf models, defined at the end of the module so everything before stays incomplete
N_MODELS = 4 # top-level models, the ones instantiated by the racing threads
N_METHODS = 60 # dummy methods to enlarge each class __dict__ (the dict iterated during annotation eval)
N_THREADS = 32
ROUNDS = 200
RACE_WINDOW_S = 0.5
def make_module_source() -> str:
boxes = " | ".join(f"Box{g}[Leaf{l}]" for g in range(N_GENERICS) for l in range(N_LEAVES))
methods = [f" def method{m}(self) -> None: ..." for m in range(N_METHODS)]
lines = ["from __future__ import annotations", "from pydantic import BaseModel", ""]
# Top-level models come first: all their field types are still undefined here,
# so their schema build is deferred until first instantiation.
for i in range(N_MODELS):
lines += [f"class Model{i}(BaseModel):", f" items: list[{boxes}] = []", *methods, ""]
for g in range(N_GENERICS):
lines += [f"class Box{g}[T](BaseModel):", " value: T | None = None", " ref: Leaf0 | None = None", *methods, ""]
for l in range(N_LEAVES):
lines += [f"class Leaf{l}(BaseModel):", " x: int = 0", ""]
return "\n".join(lines)
MODULE_SOURCE = make_module_source()
errors: list[str] = []
errors_lock = threading.Lock()
def worker(barrier: threading.Barrier, model: type) -> None:
barrier.wait()
end = time.monotonic() + RACE_WINDOW_S
while time.monotonic() < end:
try:
model() # first call hits the mock validator -> lazy model_rebuild()
except Exception:
with errors_lock:
errors.append(traceback.format_exc())
for rnd in range(ROUNDS):
# Fresh module => fresh incomplete classes, like a freshly started process.
mod = types.ModuleType(f"repro_generated_{rnd}")
sys.modules[mod.__name__] = mod
exec(compile(MODULE_SOURCE, mod.__name__, "exec"), mod.__dict__)
models = [getattr(mod, f"Model{i}") for i in range(N_MODELS)]
assert not models[0].__pydantic_complete__
barrier = threading.Barrier(N_THREADS)
threads = [threading.Thread(target=worker, args=(barrier, models[i % N_MODELS])) for i in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
hits = [tb for tb in errors if "changed size during iteration" in tb]
if hits:
print(hits[0], file=sys.stderr)
print(f"REPRODUCED in round {rnd}: RuntimeError: dictionary changed size during iteration")
sys.exit(1)
if errors:
first_line = errors[0].strip().splitlines()[-1]
print(f"round {rnd}: {len(errors)} sibling race errors (e.g. {first_line})")
errors.clear()
print("not reproduced")
Python, Pydantic & OS Version
In Prod: Ubuntu Docker, in Dev:
pydantic version: 2.13.4
pydantic-core version: 2.46.4
pydantic-core build: profile=release pgo=false
python version: 3.14.6 | packaged by conda-forge | (main, Jun 12 2026, 08:47:56) [Clang 20.1.8 ]
platform: macOS-26.5.1-arm64-arm-64bit-Mach-O
related packages: pyright-1.1.410 fastapi-0.136.3 pydantic-extra-types-2.11.1 email-validator-2.3.0 pydantic-settings-2.14.1 typing_extensions-4.15.0
commit: unknownThe two patches
pydantic/_internal/_namespace_utils.pyindex d308f7e4..691f7ea1 100644@@ -88,7 +88,17 @@ class LazyLocalNamespace(Mapping[str, Any]): @cached_property def data(self) -> dict[str, Any]:- return {k: v for ns in self._namespaces for k, v in ns.items()}+ data: dict[str, Any] = {}+ for ns in self._namespaces:+ # A namespace can be a live mapping (e.g. a class `__dict__`) that+ # another thread mutates while we read it -- this happens when a+ # `model_rebuild()` call runs concurrently for the class being+ # inspected. Iterating such a namespace directly can raise+ # `RuntimeError: dictionary changed size during iteration`, so we+ # take an atomic copy first (`dict`/`mappingproxy` copies don't call+ # back into Python and can't observe a partially-mutated state).+ data.update(ns.copy() if hasattr(ns, 'copy') else ns)+ return data def __len__(self) -> int: return len(self.data)tests/test_forward_ref.pyindex 9b0325d2..2875b12b 100644@@ -1144,6 +1144,24 @@ def test_lazy_local_namespace_len() -> None: assert len(namespace) == 2 +def test_lazy_local_namespace_data_mutated_during_iteration() -> None:+ # A namespace passed to `LazyLocalNamespace` can be a live mapping (e.g. a+ # class `__dict__`) that another thread mutates while `.data` reads it, as+ # happens when a `model_rebuild()` runs concurrently for the class being+ # inspected. `.data` must not raise `RuntimeError: dictionary changed size+ # during iteration` (concurrent model_rebuild).+ class MutatingNamespace(dict):+ def items(self): # type: ignore[override]+ for key, value in super().items():+ self[object()] = None # simulate a concurrent mutation+ yield key, value++ namespace = LazyLocalNamespace(MutatingNamespace({'a': int, 'b': str}))+ # Reading a key builds `.data`; this must succeed instead of raising.+ assert namespace['a'] is int+ assert namespace['b'] is str++ @pytest.mark.xfail( reason='While `get_cls_type_hints` uses the correct module ns for each base, `collect_model_fields` ' 'will still use the `FieldInfo` instances from each base (see the `parent_fields_lookup` logic). '
pydantic/dataclasses.pyindex 62d380f54..474289ec3 100644@@ -330,9 +330,13 @@ def rebuild_dataclass( return None for attr in ('__pydantic_core_schema__', '__pydantic_validator__', '__pydantic_serializer__'):- if attr in cls.__dict__ and not isinstance(getattr(cls, attr), _mock_val_ser.MockValSer):+ if attr in cls.__dict__ and not isinstance(+ getattr(cls, attr), (_mock_val_ser.MockCoreSchema, _mock_val_ser.MockValSer)+ ): # Deleting the validator/serializer is necessary as otherwise they can get reused in # pydantic-core. Same applies for the core schema that can be reused in schema generation.+ # We do so only if they aren't mock instances, otherwise concurrent reads of these attributes+ # (e.g. when instantiating the dataclass in another thread) can resolve them from the parent class. delattr(cls, attr) cls.__pydantic_complete__ = Falsepydantic/main.pyindex 758a839c6..6bb1554e5 100644@@ -8,6 +8,7 @@ from __future__ import annotations as _annotations import operator import sys+import threading import types import warnings from collections.abc import Generator, Mapping@@ -78,6 +79,18 @@ IncEx: TypeAlias = set[int] | set[str] | Mapping[int, Union['IncEx', bool]] | Ma _object_setattr = _model_construction.object_setattr +_rebuild_lock = threading.RLock()+"""+A lock used to make model rebuilds thread-safe when first instantiating an incomplete model.++Rebuilding a model isn't thread-safe (the class attributes are mutated during the rebuild,+while other threads may be reading them to perform validation/serialization), so `model_rebuild()`+calls are serialized using this lock. The lock is reentrant, as rebuilding a model can trigger the+rebuild of another one (e.g. when a generic origin is rebuilt during parametrization in+`__class_getitem__()`). For the same reason, the lock is global and not per-class: two threads+holding their own class's lock could otherwise request the other's and deadlock.+"""+ def _check_frozen(model_cls: type[BaseModel], name: str, value: Any) -> None: if model_cls.model_config.get('frozen'):@@ -681,42 +694,54 @@ class BaseModel(metaclass=_model_construction.ModelMetaclass): Returns `None` if the schema is already "complete" and rebuilding was not required. If rebuilding _was_ required, returns `True` if rebuilding was successful, otherwise `False`. """- already_complete = cls.__pydantic_complete__- if already_complete and not force:+ # As the rebuild lock is global (not per model class), avoid needlessly holding it if the model is+ # already complete:+ if cls.__pydantic_complete__ and not force: return None - cls.__pydantic_complete__ = False+ with _rebuild_lock:+ # Re-check inside the lock, as another thread may have rebuilt the model while we were waiting:+ already_complete = cls.__pydantic_complete__+ if already_complete and not force:+ return None - for attr in ('__pydantic_core_schema__', '__pydantic_validator__', '__pydantic_serializer__'):- if attr in cls.__dict__ and not isinstance(getattr(cls, attr), _mock_val_ser.MockValSer):- # Deleting the validator/serializer is necessary as otherwise they can get reused in- # pydantic-core. We do so only if they aren't mock instances, otherwise — as `model_rebuild()`- # isn't thread-safe — concurrent model instantiations can lead to the parent validator being used.- # Same applies for the core schema that can be reused in schema generation.- delattr(cls, attr)+ cls.__pydantic_complete__ = False - if _types_namespace is not None:- rebuild_ns = _types_namespace- elif _parent_namespace_depth > 0:- rebuild_ns = _typing_extra.parent_frame_namespace(parent_depth=_parent_namespace_depth, force=True) or {}- else:- rebuild_ns = {}+ for attr in ('__pydantic_core_schema__', '__pydantic_validator__', '__pydantic_serializer__'):+ if attr in cls.__dict__ and not isinstance(+ getattr(cls, attr), (_mock_val_ser.MockCoreSchema, _mock_val_ser.MockValSer)+ ):+ # Deleting the validator/serializer is necessary as otherwise they can get reused in+ # pydantic-core. Same applies for the core schema that can be reused in schema generation.+ # We do so only if they aren't mock instances, otherwise concurrent reads of these attributes+ # — performed without holding the rebuild lock (e.g. when instantiating the model) — can+ # resolve them from the parent class.+ delattr(cls, attr)++ if _types_namespace is not None:+ rebuild_ns = _types_namespace+ elif _parent_namespace_depth > 0:+ rebuild_ns = (+ _typing_extra.parent_frame_namespace(parent_depth=_parent_namespace_depth, force=True) or {}+ )+ else:+ rebuild_ns = {} - parent_ns = _model_construction.unpack_lenient_weakvaluedict(cls.__pydantic_parent_namespace__) or {}+ parent_ns = _model_construction.unpack_lenient_weakvaluedict(cls.__pydantic_parent_namespace__) or {} - ns_resolver = _namespace_utils.NsResolver(- parent_namespace={**rebuild_ns, **parent_ns},- )+ ns_resolver = _namespace_utils.NsResolver(+ parent_namespace={**rebuild_ns, **parent_ns},+ ) - return _model_construction.complete_model_class(- cls,- _config.ConfigWrapper(cls.model_config, check=False),- ns_resolver,- raise_errors=raise_errors,- # If the model was already complete, we don't need to call the hook again.- call_on_complete_hook=not already_complete,- is_force_rebuild=force,- )+ return _model_construction.complete_model_class(+ cls,+ _config.ConfigWrapper(cls.model_config, check=False),+ ns_resolver,+ raise_errors=raise_errors,+ # If the model was already complete, we don't need to call the hook again.+ call_on_complete_hook=not already_complete,+ is_force_rebuild=force,+ ) @classmethod def model_validate(tests/test_main.pyindex c87dae793..f924cec99 100644@@ -2,6 +2,7 @@ import json import platform import re import sys+import threading import typing import warnings from collections import defaultdict@@ -2541,6 +2542,84 @@ def test_model_rebuild_localns(): C.model_rebuild(_types_namespace={'A': A}) +@pytest.mark.timeout(5)+def test_model_rebuild_nested_during_parametrization():+ T = TypeVar('T')++ class Model(BaseModel):+ b: Optional['Box[Leaf]'] = None++ class Box(BaseModel, Generic[T]):+ leaf: Optional['Leaf'] = None++ class Leaf(BaseModel):+ x: int = 0++ assert not Model.__pydantic_complete__+ assert not Box.__pydantic_complete__++ # Rebuilding `Model` evaluates `Box[Leaf]`, and `__class_getitem__` then calls+ # `Box.model_rebuild()` on the same thread. Any locking added to the rebuild logic+ # must support this reentrancy:+ m = Model(b={'leaf': {'x': 1}})+ assert m.b.leaf.x == 1+ assert Model.__pydantic_complete__+++@pytest.mark.skipif(sys.platform == 'emscripten', reason='no threading on emscripten')+def test_model_rebuild_already_rebuilt_by_other_thread():+ schema_gen_entered = threading.Event()+ resume_schema_gen = threading.Event()+ hook_calls: list[type] = []++ class Model(BaseModel):+ x: 'Gate'++ @classmethod+ def __pydantic_on_complete__(cls) -> None:+ hook_calls.append(cls)++ class Gate:+ @classmethod+ def __get_pydantic_core_schema__(cls, source_type, handler):+ schema_gen_entered.set()+ resume_schema_gen.wait(timeout=10)+ return core_schema.int_schema()++ assert not Model.__pydantic_complete__++ results: dict[str, bool | None] = {}+ t2_started = threading.Event()++ def rebuild(key: str) -> None:+ if key == 't2':+ t2_started.set()+ results[key] = Model.model_rebuild(_types_namespace={'Gate': Gate})++ # The first thread acquires the rebuild lock and waits in schema generation:+ t1 = threading.Thread(target=rebuild, args=('t1',), daemon=True)+ t1.start()+ assert schema_gen_entered.wait(timeout=10)++ # While the first thread is waiting on `resume_schema_gen`, `Model` can't become complete,+ # so the second thread is guaranteed to pass the unlocked completeness check and block on the lock:+ t2 = threading.Thread(target=rebuild, args=('t2',), daemon=True)+ t2.start()+ assert t2_started.wait(timeout=10)+ assert t2.is_alive()+ assert 't2' not in results++ resume_schema_gen.set()+ t1.join(timeout=10)+ t2.join(timeout=10)+ assert not t1.is_alive()+ assert not t2.is_alive()++ assert results == {'t1': True, 't2': None}+ assert hook_calls == [Model]+ assert Model(x=1).x == 1++ def test_model_rebuild_zero_depth(): class Model(BaseModel): x: 'X_Type'
What the agent reported
The work is complete. Here is my final report.
What was wrong
Concurrent lazy model_rebuild() calls raised RuntimeError: dictionary changed size during iteration in LazyLocalNamespace.data (pydantic/_internal/_namespace_utils.py).
NsResolver.types_namespace appends the live class __dict__ (vars(typ), a mappingproxy) into a LazyLocalNamespace (line 276). LazyLocalNamespace.data then iterated that mapping live via a dict comprehension over ns.items(). When a second thread ran model_rebuild() for the same class — deleting/setting __pydantic_core_schema__, __pydantic_validator__, __pydantic_serializer__ on the class __dict__ — the in-flight iteration observed the size change and crashed. FastAPI's threadpool serving the first requests after process start is a natural trigger.
What I changed
pydantic/_internal/_namespace_utils.py — rewrote LazyLocalNamespace.data to copy each namespace before merging instead of iterating it live:
data: dict[str, Any] = {}
for ns in self._namespaces:
data.update(ns.copy() if hasattr(ns, 'copy') else ns)
return data
dict.copy() / mappingproxy.copy() are atomic C-level operations that don't call back into Python and can't observe a partially-mutated state (I verified mappingproxy.copy() delegates to the underlying dict.copy(), and a dict subclass's .copy() returns a plain dict without invoking any overridden items()). Merge semantics are unchanged — later namespaces still override earlier ones.
tests/test_forward_ref.py — added test_lazy_local_namespace_data_mutated_during_iteration, which deterministically (no threads) reproduces the failure using a namespace whose items() mutates itself mid-iteration, then asserts .data still resolves keys.
Test evidence
- New test fails before the fix with the exact
RuntimeError: dictionary changed size during iteration; passes after. ./bt-test tests/test_forward_ref.py::test_lazy_local_namespace_data_mutated_during_iteration tests/test_forward_ref.py::test_lazy_local_namespace_len→ 2 passed.- Full
tests/test_forward_ref.py→ 68 passed, 4 pre-existing xfailed. - Blast radius
tests/test_internal.py tests/test_construction.py→ 81 passed, 2 pre-existing xfailed. - Reviewer confirmed the fix resolves the reported bug, preserves merge/len semantics, changes nothing unrelated, and the test genuinely distinguishes old vs. new behavior.
Requirements disposition
- Fix
RuntimeError: dictionary changed size during iterationduring concurrentmodel_rebuild— ✅ Implemented and covered by the new regression test (fails before, passes after). - Reproducer script provided in report — ✅ Its failure mode is captured deterministically by the new unit test. I did not run the flaky, timing-dependent 200-round threaded script as a gate (it targets 3.14 and can take up to a minute per run); the deterministic test exercises the identical code path and mutation-during-iteration condition.
- The related intermittent
AttributeError: ... has no attribute '__pydantic_core_schema__'(from the delattr block inmodel_rebuild) — mentioned in the report only as a secondary symptom of the same race, not requested to be fixed. It originates in a different location (main.py/_model_construction.py, not_namespace_utils.py) and is out of scope for thisLazyLocalNamespace.datafix; I did not change it.
All changes are left uncommitted in the working tree.