Backtestspydantic-airun-20260918-215936#7629

Issue #7629 Merged fix PR # Run 18 Sep 2026 · run-20260918-215936 pydantic/pydantic-ai

TestModel generates values that violate falsy JSON Schema const constraints

Worse

The merged fix was judged better than our patch.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches apply the identical, correct root-cause fix: replacing the truthiness check `if const := schema.get('const')` with a membership test `if 'const' in schema`, which is exactly the direction the issue suggests. So correctness and scope of the source change are a wash. The decision turns on test quality. Patch A adds one focused unit test calling `_JsonSchemaTestData` directly with the three falsy-const cases — clean, minimal, and it fails before / passes after. Patch B does more and better: it extends the existing `test_json_schema_test_data` snapshot with `Literal['']`, `Literal[False]`, `Literal[0]` fields (single-value Literals compile to JSON Schema `const` in pydantic, so these genuinely exercise the fixed path within the established regression fixture), AND adds an end-to-end `test_falsy_const_tool_args` that drives a real `Agent` + `TestModel` with falsy-const tool arguments, asserting the tool receives `''`/`False`/`0`. That integration test mirrors the exact user-facing symptom the issue calls out ('generate tool arguments ... that violate the schema'), which A's internal-only test does not cover. Both B tests would fail pre-fix and pass post-fix. Minor dings against B: it invents an issue number (#7629) not present in the report, and it touches slightly more surface — but the added coverage is justified and directly tied to the reported failure mode. Net: equal fix, but B's tests are more thorough and closer to the real-world defect, so B is the better patch.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

10m 33s1 turn

Wall-clock time in the container

Full test suite

No new failures

11,724 passed · 574 skipped · 12,543 total

Contamination audit

Unknown

See the audit notes for this run

Patch size

18vs 26 merged

Changed lines, ours vs merged

TestModel generates values that violate falsy JSON Schema const constraints

Initial Checks

  • I reproduced this against the current main checkout at 5976ebf1038368ad16ca3507a334b06c8fb6f2ea in a tokenless Python 3.12 container.
  • I also reproduced it against released pydantic-ai v2.29.0 (f748e81ec64aff78c4d41593ed51803d8b15b9ee).
  • I searched open and closed issues and pull requests for TestModel/const, _JsonSchemaTestData, and falsy JSON Schema const; I found no report of this TestModel generator behavior. I also checked #3610, which concerns GoogleJsonSchemaTransformer, not _JsonSchemaTestData.

Description

TestModel's JSON Schema data generator checks const by truthiness:

if const := schema.get('const'):
    return const

Valid JSON Schema constants such as "", False, and 0 therefore fall through to ordinary type generation. This can make TestModel generate tool arguments or structured-output values that violate the schema it was given, producing misleading test behavior or validation retries.

Minimal, Reproducible Example

This uses the internal generator directly to keep the reproducer small:

from pydantic_ai.models.test import _JsonSchemaTestData

cases = {
    'empty_string': {'type': 'string', 'const': ''},
    'false': {'type': 'boolean', 'const': False},
    'zero': {'type': 'integer', 'const': 0},
}

for name, schema in cases.items():
    value = _JsonSchemaTestData(
        {
            'type': 'object',
            'required': ['value'],
            'properties': {'value': schema},
        },
        seed=1,
    ).generate()['value']
    print(name, repr(value))

Observed with released pydantic-ai v2.29.0:

empty_string 'b'
false True
zero 1

Expected output is '', False, and 0, respectively.

Suggested Direction

Distinguish an absent const from a present falsy value (for example, test for the key rather than the value's truthiness), then add focused coverage for these constants.

Environment

  • Python: 3.12
  • Pydantic AI: released v2.29.0 (f748e81ec64aff78c4d41593ed51803d8b15b9ee)
  • LLM provider SDK: not applicable; this is a local TestModel schema-generation path

The two patches

Our patch+16 −2
pydantic_ai_slim/pydantic_ai/models/test.pyindex 65ecb38e..3d758925 100644@@ -460,8 +460,8 @@ class _JsonSchemaTestData:      def _gen_any(self, schema: dict[str, Any]) -> Any:         """Generate data for any JSON Schema."""-        if const := schema.get('const'):-            return const+        if 'const' in schema:+            return schema['const']         elif enum := schema.get('enum'):             return enum[self.seed % len(enum)]         elif examples := schema.get('examples'):tests/models/test_model_test.pyindex cb45fb55..82d1d3aa 100644@@ -569,6 +569,20 @@ def test_json_schema_test_data_equal_inclusive_bounds():     TestModel.model_validate(data)  +def test_json_schema_test_data_falsy_const():+    schema = {+        'type': 'object',+        'required': ['empty_string', 'false', 'zero'],+        'properties': {+            'empty_string': {'type': 'string', 'const': ''},+            'false': {'type': 'boolean', 'const': False},+            'zero': {'type': 'integer', 'const': 0},+        },+    }+    data = _JsonSchemaTestData(schema, seed=1).generate()+    assert data == snapshot({'empty_string': '', 'false': False, 'zero': 0})++ def test_chars_wrap():     class TestModel(BaseModel):         a: Annotated[set[str], MinLen(4)] 
Merged fix · PR #+24 −2
pydantic_ai_slim/pydantic_ai/models/test.pyindex 65ecb38e09..3d7589253b 100644@@ -460,8 +460,8 @@ class _JsonSchemaTestData:      def _gen_any(self, schema: dict[str, Any]) -> Any:         """Generate data for any JSON Schema."""-        if const := schema.get('const'):-            return const+        if 'const' in schema:+            return schema['const']         elif enum := schema.get('enum'):             return enum[self.seed % len(enum)]         elif examples := schema.get('examples'):tests/models/test_model_test.pyindex cb45fb5511..7b1da8ba33 100644@@ -497,6 +497,9 @@ def test_json_schema_test_data():         my_lit_ints: Literal[1, 2, 3]         my_lit_str: Literal['a']         my_lit_strs: Literal['a', 'b', 'c']+        my_lit_empty: Literal['']+        my_lit_false: Literal[False]+        my_lit_zero: Literal[0]         my_any: Any         nested: NestedModel         union: int | list[int]@@ -535,6 +538,9 @@ def test_json_schema_test_data():             'my_lit_ints': 1,             'my_lit_str': 'a',             'my_lit_strs': 'a',+            'my_lit_empty': '',+            'my_lit_false': False,+            'my_lit_zero': 0,             'my_any': 'g',             'union': 6,             'optional': 'g',@@ -598,6 +604,22 @@ def test_max_items():     assert data == snapshot([])  +def test_falsy_const_tool_args() -> None:+    """Regression test for #7629: falsy JSON Schema `const` values must be generated as-is."""++    agent = Agent()+    calls: list[dict[str, Any]] = []++    @agent.tool_plain+    def my_tool(empty: Literal[''], flag: Literal[False], zero: Literal[0]) -> str:+        calls.append({'empty': empty, 'flag': flag, 'zero': zero})+        return 'ok'++    result = agent.run_sync('hello', model=TestModel())+    assert result.output == snapshot('{"my_tool":"ok"}')+    assert calls == snapshot([{'empty': '', 'flag': False, 'zero': 0}])++ @pytest.mark.parametrize(     'content',     [ 

What the agent reported

Reviewer approved. The work is complete.

Summary

What was wrong: TestModel's JSON Schema data generator _JsonSchemaTestData._gen_any (pydantic_ai_slim/pydantic_ai/models/test.py:463) checked const by truthiness:

if const := schema.get('const'):
    return const

Valid falsy constants ('', False, 0) failed this check and fell through to ordinary type generation, so TestModel produced values that violated the schema's const — e.g. 'b', True, 1 instead of '', False, 0.

What I changed:

  • pydantic_ai_slim/pydantic_ai/models/test.py (production, 2 lines) — replaced the truthiness check with a key-presence check so a present falsy const is returned verbatim: python if 'const' in schema: return schema['const'] Only the const branch changed; enum/examples/anyOf were left as-is (out of scope for this report).
  • tests/models/test_model_test.py (regression test) — added test_json_schema_test_data_falsy_const, mirroring the established raw-dict-schema convention (test_prefix_unique, test_max_items) and the report's reproducer. It generates for a schema with const: '', const: False, const: 0 and asserts {'empty_string': '', 'false': False, 'zero': 0}.

Test evidence:

  • Before the fix (isolated run): test_json_schema_test_data_falsy_const FAILED — generator returned {'empty_string': 'b', 'false': True, 'zero': 1}, matching the report.
  • After the fix: full tests/models/test_model_test.py — 26 passed, including the new regression test and all pre-existing _JsonSchemaTestData/TestModel tests.
  • Independent reviewer verdict: APPROVE (fix correct and minimal, test exercises the exact scenario, no existing tests weakened).

Requirements checklist:

  1. Distinguish absent const from present falsy const (report's suggested direction) — Implemented; key-presence check, verified by passing regression test.
  2. Cover falsy constants '', False, 0 (report's three named cases) — Implemented; all three asserted in test_json_schema_test_data_falsy_const, seen failing before and passing after.
  3. Keep behavior for non-falsy const/enum/examples intact — Verified; those branches unchanged and the full test_model_test.py suite (which exercises Literal[1] const and enum paths) still passes.

All changes are left uncommitted in the working tree.