Backtestspydanticrun-20261027-165153#13729
Smart union selects the wrong model in JSON mode with a before validator
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The Rust change is byte-for-byte equivalent in both patches (apart from one blank line): both remove the outer `state.add_fields_set(fields_set.len())` in `ModelFieldsValidator::validate` and instead count successfully validated known fields inside `validate_json_object`, adding that count once at the end. That matches the issue's diagnosis: the Python-input helper already added its own count, so the outer add double-counted on the before-validator path, while the plain JSON path got only the outer count. Independent tests confirm both patches fix every scenario, including reversed union order and nested-field usage. The difference is tests. Patch A adds a regression test at the pydantic-core level (tests/validators/test_union.py) using a raw `no_info_before_validator_function` wrapping a `model_schema`, parametrized over python/json and over permuted choice order, which exercises the fix where it actually lives and would catch a regression in pydantic-core independently of the pydantic wrapper. It also adds an end-to-end reproduction of the issue example in tests/test_validators.py. Patch B adds only pydantic-level tests, albeit with extra/reverse-order parametrization, and leaves an unfilled `<link>` docstring placeholder (stripped reference, ignored for scoring but still a stub). Both patches include an extra-fields hardening test that is neutral. Scope and craft are otherwise equal, so Patch A's two-layer test coverage, matched to the location of the fix, gives it a modest but real edge.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
7m 28s
Wall-clock time in the container
Full test suite
No new failures
488 passed · 92 skipped · 580 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
103vs 55 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I confirm that I'm using Pydantic V2
Description
Adding a no-op mode='before' model validator makes smart-union validation select a model with fewer matching fields for JSON input. The equivalent Python input selects the model with more matching fields.
The example below prints:
Larger
Smaller
I expect Larger in both cases: it has three supplied valid fields, while Smaller has two. This follows the documented number-of-valid-fields-set ranking.
Reproduced on main at 965c23dd93bd5ca7b86224ba39ccbe79399f117b. In model_fields.rs, the Python-input helper adds its known-field count, then the outer validator adds fields_set.len() again. The before validator sends the smaller model through that Python-input path, while the larger model uses the plain JSON path and receives only the outer count.
I can send a focused fix with regression tests. Could you assign this to me?
Example Code
from typing import Any
from pydantic import BaseModel, TypeAdapter, model_validator
class Smaller(BaseModel):
a: int
b: int
@model_validator(mode='before')
@classmethod
def keep_input(cls, value: Any) -> Any:
return value
class Larger(BaseModel):
a: int
b: int
c: int
adapter = TypeAdapter(Smaller | Larger)
print(type(adapter.validate_python({'a': 1, 'b': 2, 'c': 3})).__name__)
print(type(adapter.validate_json('{"a": 1, "b": 2, "c": 3}')).__name__)
Python, Pydantic & OS Version
pydantic version: 2.14.0b1
pydantic-core version: 2.48.0
pydantic-core build: profile=debug pgo=false
python version: 3.13.14 (main, Jul 18 2026, 16:56:45) [Clang 22.1.3 ]
platform: macOS-26.6.2-arm64-arm-64bit-Mach-O
related packages: mypy-2.3.0 pyright-1.1.411 typing_extensions-4.15.0
commit: 965c23dd9
Prepared with AI assistance.
The two patches
pydantic-core/src/validators/model_fields.rsindex 5e8fe034..c0dcd5e7 100644@@ -179,7 +179,6 @@ impl Validator for ModelFieldsValidator { }; self.validate_by_get_item(py, input, dict, state)? };- state.add_fields_set(fields_set.len()); // if we have extra=allow, but we didn't create a dict because we were validating // from attributes, set it now so __pydantic_extra__ is always a dict if extra=allow@@ -560,6 +559,7 @@ impl ModelFieldsValidator { (0..self.fields.len()).map(|_| None).collect(); let mut errors: Vec<ValLineError> = Vec::new(); let fields_set = PySet::empty(py)?;+ let mut fields_set_count: usize = 0; let state = &mut state.scoped_set_data(Some(model_dict.clone())); let state = &mut state.scoped_clear_field_error();@@ -636,6 +636,7 @@ impl ModelFieldsValidator { match field.validator.validate(py, field_json_value, state) { Ok(value) => { fields_set.add(&field.name)?;+ fields_set_count += 1; value } Err(ValError::Omit) => continue,@@ -697,6 +698,8 @@ impl ModelFieldsValidator { return Err(ValError::LineErrors(errors)); } + state.add_fields_set(fields_set_count);+ Ok((model_dict, model_extra_dict_op, fields_set)) } }pydantic-core/tests/validators/test_union.pyindex 268e5a40..4a49ec3b 100644@@ -1424,3 +1424,79 @@ def test_smart_union_wrap_validator_should_not_change_nested_model_field_counts( assert isinstance(m, RootModel) assert isinstance(m.ab, ModelA) assert m.ab.sub.x == 'y'+++@pytest.mark.parametrize('mode', ['python', 'json'])+def test_smart_union_before_validator_does_not_change_model_field_counts(mode) -> None:+ """A before validator on one choice must not change how model fields are counted for that choice."""++ class Smaller:+ a: int+ b: int++ class Larger:+ a: int+ b: int+ c: int++ smaller_schema = core_schema.no_info_before_validator_function(+ lambda v: v,+ core_schema.model_schema(+ Smaller,+ core_schema.model_fields_schema(+ fields={+ 'a': core_schema.model_field(core_schema.int_schema()),+ 'b': core_schema.model_field(core_schema.int_schema()),+ }+ ),+ ),+ )+ larger_schema = core_schema.model_schema(+ Larger,+ core_schema.model_fields_schema(+ fields={+ 'a': core_schema.model_field(core_schema.int_schema()),+ 'b': core_schema.model_field(core_schema.int_schema()),+ 'c': core_schema.model_field(core_schema.int_schema()),+ }+ ),+ )++ for choices in permute_choices([smaller_schema, larger_schema]):+ validator = SchemaValidator(core_schema.union_schema(choices))+ if mode == 'python':+ result = validator.validate_python({'a': 1, 'b': 2, 'c': 3})+ else:+ result = validator.validate_json('{"a": 1, "b": 2, "c": 3}')+ assert isinstance(result, Larger)+++@pytest.mark.parametrize('extra_behavior', ['forbid', 'ignore', 'allow'])+def test_smart_union_extra_behavior_json(extra_behavior) -> None:+ """Extra fields must not count as valid fields set in JSON mode, matching Python mode."""++ class Foo:+ foo: str = 'foo'++ class Bar:+ bar: str = 'bar'++ def member_schema(cls, field_name):+ return core_schema.model_schema(+ cls,+ core_schema.model_fields_schema(+ fields={+ field_name: core_schema.model_field(+ core_schema.with_default_schema(core_schema.str_schema(), default=field_name)+ )+ }+ ),+ extra_behavior=extra_behavior,+ )++ validator = SchemaValidator(core_schema.union_schema([member_schema(Foo, 'foo'), member_schema(Bar, 'bar')]))++ assert isinstance(validator.validate_json('{"foo": "foo"}'), Foo)+ assert isinstance(validator.validate_json('{"bar": "bar"}'), Bar)+ assert isinstance(validator.validate_python({'foo': 'foo'}), Foo)+ assert isinstance(validator.validate_python({'bar': 'bar'}), Bar)tests/test_validators.pyindex 7e9f4713..f6352eb9 100644@@ -3254,3 +3254,25 @@ def test_wrap_validator_forwards_runtime_settings() -> None: with pytest.raises(ValidationError, match='Input should be a valid dictionary'): Outer.model_validate({'inner': InnerObj()}, context='ctx') assert Outer.model_validate({'inner': InnerObj()}, from_attributes=True, context='ctx').inner.a == 1+++def test_smart_union_model_before_validator_json_and_python() -> None:+ """A `mode='before'` model validator must not change smart union field-count ranking in JSON mode."""++ class Smaller(BaseModel):+ a: int+ b: int++ @model_validator(mode='before')+ @classmethod+ def keep_input(cls, value: Any) -> Any:+ return value++ class Larger(BaseModel):+ a: int+ b: int+ c: int++ adapter = TypeAdapter(Smaller | Larger)+ assert type(adapter.validate_python({'a': 1, 'b': 2, 'c': 3})) is Larger+ assert type(adapter.validate_json('{"a": 1, "b": 2, "c": 3}')) is Larger
pydantic-core/src/validators/model_fields.rsindex 5e8fe0349..913d51292 100644@@ -179,7 +179,6 @@ impl Validator for ModelFieldsValidator { }; self.validate_by_get_item(py, input, dict, state)? };- state.add_fields_set(fields_set.len()); // if we have extra=allow, but we didn't create a dict because we were validating // from attributes, set it now so __pydantic_extra__ is always a dict if extra=allow@@ -560,6 +559,7 @@ impl ModelFieldsValidator { (0..self.fields.len()).map(|_| None).collect(); let mut errors: Vec<ValLineError> = Vec::new(); let fields_set = PySet::empty(py)?;+ let mut fields_set_count: usize = 0; let state = &mut state.scoped_set_data(Some(model_dict.clone())); let state = &mut state.scoped_clear_field_error();@@ -636,6 +636,7 @@ impl ModelFieldsValidator { match field.validator.validate(py, field_json_value, state) { Ok(value) => { fields_set.add(&field.name)?;+ fields_set_count += 1; value } Err(ValError::Omit) => continue,@@ -697,6 +698,7 @@ impl ModelFieldsValidator { return Err(ValError::LineErrors(errors)); } + state.add_fields_set(fields_set_count); Ok((model_dict, model_extra_dict_op, fields_set)) } }tests/types/unions/test_union.pyindex b68d9cf19..f7a3f60f2 100644@@ -1,6 +1,6 @@ from abc import ABC, abstractmethod from enum import IntEnum-from typing import Annotated, ClassVar, Literal+from typing import Annotated, Any, ClassVar, Literal from uuid import UUID import pytest@@ -14,6 +14,7 @@ from pydantic import ( Strict, TypeAdapter, ValidationError,+ model_validator, ) @@ -235,6 +236,54 @@ def test_union_typeddict(): assert ta.validate_python(dict(foo='baz')) == {'foo': 'baz'} +@pytest.mark.parametrize('extra', ['ignore', 'allow'])+@pytest.mark.parametrize('reverse_members', [False, True])+def test_smart_union_fields_set_count_consistent_between_json_and_python(+ extra: Literal['ignore', 'allow'], reverse_members: bool+) -> None:+ """https://github.com/pydantic/pydantic/issues/13729"""++ class Smaller(BaseModel):+ model_config = ConfigDict(extra=extra)++ a: int+ b: int++ @model_validator(mode='before')+ @classmethod+ def keep_input(cls, value: Any) -> Any:+ return value++ class Larger(BaseModel):+ a: int+ b: int+ c: int++ ta = TypeAdapter(Larger | Smaller if reverse_members else Smaller | Larger)++ assert isinstance(ta.validate_python({'a': 1, 'b': 2, 'c': 3}), Larger)+ assert isinstance(ta.validate_json('{"a": 1, "b": 2, "c": 3}'), Larger)+++@pytest.mark.parametrize('reverse_members', [False, True])+def test_smart_union_fields_set_count_ignores_extra(reverse_members: bool) -> None:+ """https://github.com/pydantic/pydantic/issues/13729"""++ class WithExtra(BaseModel):+ model_config = ConfigDict(extra='allow')++ a: int++ class Explicit(BaseModel):+ a: int+ b: int++ ta = TypeAdapter(Explicit | WithExtra if reverse_members else WithExtra | Explicit)++ assert isinstance(ta.validate_python({'a': 1, 'b': 2}), Explicit)+ assert isinstance(ta.validate_json('{"a": 1, "b": 2}'), Explicit)++ def test_union_respects_local_strict() -> None: class MyBaseModel(BaseModel): model_config = ConfigDict(strict=True)