Backtestspydantic-airun-20260918-215936#7629
TestModel generates values that violate falsy JSON Schema const constraints
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches apply the identical, correct root-cause fix: replacing the truthiness check `if const := schema.get('const')` with a membership test `if 'const' in schema`, which is exactly the direction the issue suggests. So correctness and scope of the source change are a wash. The decision turns on test quality. Patch A adds one focused unit test calling `_JsonSchemaTestData` directly with the three falsy-const cases — clean, minimal, and it fails before / passes after. Patch B does more and better: it extends the existing `test_json_schema_test_data` snapshot with `Literal['']`, `Literal[False]`, `Literal[0]` fields (single-value Literals compile to JSON Schema `const` in pydantic, so these genuinely exercise the fixed path within the established regression fixture), AND adds an end-to-end `test_falsy_const_tool_args` that drives a real `Agent` + `TestModel` with falsy-const tool arguments, asserting the tool receives `''`/`False`/`0`. That integration test mirrors the exact user-facing symptom the issue calls out ('generate tool arguments ... that violate the schema'), which A's internal-only test does not cover. Both B tests would fail pre-fix and pass post-fix. Minor dings against B: it invents an issue number (#7629) not present in the report, and it touches slightly more surface — but the added coverage is justified and directly tied to the reported failure mode. Net: equal fix, but B's tests are more thorough and closer to the real-world defect, so B is the better patch.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
10m 33s1 turn
Wall-clock time in the container
Full test suite
No new failures
11,724 passed · 574 skipped · 12,543 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
18vs 26 merged
Changed lines, ours vs merged
The issue
View on GitHubTestModel generates values that violate falsy JSON Schema const constraints
Initial Checks
- I reproduced this against the current
maincheckout at5976ebf1038368ad16ca3507a334b06c8fb6f2eain a tokenless Python 3.12 container. - I also reproduced it against released
pydantic-aiv2.29.0(f748e81ec64aff78c4d41593ed51803d8b15b9ee). - I searched open and closed issues and pull requests for
TestModel/const,_JsonSchemaTestData, andfalsy JSON Schema const; I found no report of thisTestModelgenerator behavior. I also checked #3610, which concernsGoogleJsonSchemaTransformer, not_JsonSchemaTestData.
Description
TestModel's JSON Schema data generator checks const by truthiness:
if const := schema.get('const'):
return const
Valid JSON Schema constants such as "", False, and 0 therefore fall through to ordinary type generation. This can make TestModel generate tool arguments or structured-output values that violate the schema it was given, producing misleading test behavior or validation retries.
Minimal, Reproducible Example
This uses the internal generator directly to keep the reproducer small:
from pydantic_ai.models.test import _JsonSchemaTestData
cases = {
'empty_string': {'type': 'string', 'const': ''},
'false': {'type': 'boolean', 'const': False},
'zero': {'type': 'integer', 'const': 0},
}
for name, schema in cases.items():
value = _JsonSchemaTestData(
{
'type': 'object',
'required': ['value'],
'properties': {'value': schema},
},
seed=1,
).generate()['value']
print(name, repr(value))
Observed with released pydantic-ai v2.29.0:
empty_string 'b'
false True
zero 1
Expected output is '', False, and 0, respectively.
Suggested Direction
Distinguish an absent const from a present falsy value (for example, test for the key rather than the value's truthiness), then add focused coverage for these constants.
Environment
- Python: 3.12
- Pydantic AI: released
v2.29.0(f748e81ec64aff78c4d41593ed51803d8b15b9ee) - LLM provider SDK: not applicable; this is a local
TestModelschema-generation path
The two patches
pydantic_ai_slim/pydantic_ai/models/test.pyindex 65ecb38e..3d758925 100644@@ -460,8 +460,8 @@ class _JsonSchemaTestData: def _gen_any(self, schema: dict[str, Any]) -> Any: """Generate data for any JSON Schema."""- if const := schema.get('const'):- return const+ if 'const' in schema:+ return schema['const'] elif enum := schema.get('enum'): return enum[self.seed % len(enum)] elif examples := schema.get('examples'):tests/models/test_model_test.pyindex cb45fb55..82d1d3aa 100644@@ -569,6 +569,20 @@ def test_json_schema_test_data_equal_inclusive_bounds(): TestModel.model_validate(data) +def test_json_schema_test_data_falsy_const():+ schema = {+ 'type': 'object',+ 'required': ['empty_string', 'false', 'zero'],+ 'properties': {+ 'empty_string': {'type': 'string', 'const': ''},+ 'false': {'type': 'boolean', 'const': False},+ 'zero': {'type': 'integer', 'const': 0},+ },+ }+ data = _JsonSchemaTestData(schema, seed=1).generate()+ assert data == snapshot({'empty_string': '', 'false': False, 'zero': 0})++ def test_chars_wrap(): class TestModel(BaseModel): a: Annotated[set[str], MinLen(4)]
pydantic_ai_slim/pydantic_ai/models/test.pyindex 65ecb38e09..3d7589253b 100644@@ -460,8 +460,8 @@ class _JsonSchemaTestData: def _gen_any(self, schema: dict[str, Any]) -> Any: """Generate data for any JSON Schema."""- if const := schema.get('const'):- return const+ if 'const' in schema:+ return schema['const'] elif enum := schema.get('enum'): return enum[self.seed % len(enum)] elif examples := schema.get('examples'):tests/models/test_model_test.pyindex cb45fb5511..7b1da8ba33 100644@@ -497,6 +497,9 @@ def test_json_schema_test_data(): my_lit_ints: Literal[1, 2, 3] my_lit_str: Literal['a'] my_lit_strs: Literal['a', 'b', 'c']+ my_lit_empty: Literal['']+ my_lit_false: Literal[False]+ my_lit_zero: Literal[0] my_any: Any nested: NestedModel union: int | list[int]@@ -535,6 +538,9 @@ def test_json_schema_test_data(): 'my_lit_ints': 1, 'my_lit_str': 'a', 'my_lit_strs': 'a',+ 'my_lit_empty': '',+ 'my_lit_false': False,+ 'my_lit_zero': 0, 'my_any': 'g', 'union': 6, 'optional': 'g',@@ -598,6 +604,22 @@ def test_max_items(): assert data == snapshot([]) +def test_falsy_const_tool_args() -> None:+ """Regression test for #7629: falsy JSON Schema `const` values must be generated as-is."""++ agent = Agent()+ calls: list[dict[str, Any]] = []++ @agent.tool_plain+ def my_tool(empty: Literal[''], flag: Literal[False], zero: Literal[0]) -> str:+ calls.append({'empty': empty, 'flag': flag, 'zero': zero})+ return 'ok'++ result = agent.run_sync('hello', model=TestModel())+ assert result.output == snapshot('{"my_tool":"ok"}')+ assert calls == snapshot([{'empty': '', 'flag': False, 'zero': 0}])++ @pytest.mark.parametrize( 'content', [
What the agent reported
Reviewer approved. The work is complete.
Summary
What was wrong: TestModel's JSON Schema data generator _JsonSchemaTestData._gen_any (pydantic_ai_slim/pydantic_ai/models/test.py:463) checked const by truthiness:
if const := schema.get('const'):
return const
Valid falsy constants ('', False, 0) failed this check and fell through to ordinary type generation, so TestModel produced values that violated the schema's const — e.g. 'b', True, 1 instead of '', False, 0.
What I changed:
pydantic_ai_slim/pydantic_ai/models/test.py(production, 2 lines) — replaced the truthiness check with a key-presence check so a present falsyconstis returned verbatim:python if 'const' in schema: return schema['const']Only theconstbranch changed;enum/examples/anyOfwere left as-is (out of scope for this report).tests/models/test_model_test.py(regression test) — addedtest_json_schema_test_data_falsy_const, mirroring the established raw-dict-schema convention (test_prefix_unique,test_max_items) and the report's reproducer. It generates for a schema withconst: '',const: False,const: 0and asserts{'empty_string': '', 'false': False, 'zero': 0}.
Test evidence:
- Before the fix (isolated run):
test_json_schema_test_data_falsy_constFAILED — generator returned{'empty_string': 'b', 'false': True, 'zero': 1}, matching the report. - After the fix: full
tests/models/test_model_test.py— 26 passed, including the new regression test and all pre-existing_JsonSchemaTestData/TestModeltests. - Independent reviewer verdict: APPROVE (fix correct and minimal, test exercises the exact scenario, no existing tests weakened).
Requirements checklist:
- Distinguish absent
constfrom present falsyconst(report's suggested direction) — Implemented; key-presence check, verified by passing regression test. - Cover falsy constants
'',False,0(report's three named cases) — Implemented; all three asserted intest_json_schema_test_data_falsy_const, seen failing before and passing after. - Keep behavior for non-falsy
const/enum/examplesintact — Verified; those branches unchanged and the fulltest_model_test.pysuite (which exercisesLiteral[1]const and enum paths) still passes.
All changes are left uncommitted in the working tree.