Backtestspydanticrun-20260930-144322#13089
Decimal JSON schema pattern rejects pydantic's own scientific notation serialization output (v2.12 regression)
The merged fix was judged better than our patch.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Patch A is the better fix, mainly on correctness. Patch B changes only the unconstrained branch (Case 4) of the decimal pattern by adding an optional exponent. That fixes the exact example in the issue. But the same root cause remains in every constrained branch (`max_digits` and/or `decimal_places`). For example, a `Decimal` with `max_digits=10, decimal_places=8` holding `Decimal('0.0000001')` still serializes as `'1E-7'` and still fails its own serialization-mode pattern. B's pattern also ignores `allow_inf_nan`, so `'Infinity'` and `'NaN'` output still fails validation against it. Patch A removes the pattern by default, restoring the v2.11 behavior that downstream consumers relied on, so no constrained or unconstrained value can be rejected. It also provides an opt-in `get_decimal_pattern`/`build_decimal_pattern` hook with mode-aware patterns. The serialization pattern models `str(Decimal)` output precisely, including scientific notation within the constraints, `0E-n`, and inf/nan. Patch A's tests are also much stronger: they round-trip values through `dump_json` and check them against the generated serialization pattern for several constraint combinations, test rejections, and cover `allow_inf_nan` and the config-versus-schema precedence. Patch B's tests only cover the unconstrained case. Against A: it is a larger behavioral change (dropping the pattern by default), it has some unrelated churn (moving `_HashableJsonValue`), and its docs contain placeholder `<version>` markers. These are scope and craft concerns, but they don't outweigh B leaving the reported bug class unfixed for constrained decimals.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
4m 54s15 turns
Wall-clock time in the container
Full test suite
No new failures
12,629 passed · 509 skipped · 13,138 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
68vs 573 merged
Changed lines, ours vs merged
The issue
View on GitHubInitial Checks
- I confirm that I'm using Pydantic V2
Description
model_json_schema(mode="serialization") pattern for Decimal breaks on scientific notation
The serialization schema for Decimal fields generates a regex that rejects values that model_dump_json() itself produces.
Specifically: pydantic-core's serializer calls str() on Decimal objects, which outputs scientific notation for small/large values. E.g. Decimal('0.0000001') becomes "1E-7". But the pattern added in #11987 (^(?!^[-+.]*$)[+-]?0*\d*\.?\d*$) has no [eE] component, so it'll never match those strings.
As a result, any downstream consumer validating model_dump_json() output against model_json_schema(), such as OpenAPI validators and codegen + validation tools like Orval + Zod, will intermittently reject perfectly valid serialized values. Intermittent because only certain Decimal values trigger scientific notation.
v2.11:
{"anyOf": [{"type": "string"}, {"type": "null"}]}
v2.12:
{"anyOf": [{"pattern": "^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$", "type": "string"}, {"type": "null"}]}
Two possible fixes:
- Append
(?:[eE][+-]?\d+)?to the pattern in serialization mode - Normalize
Decimalto fixed-point during serialization before it ever hits the schema
Example Code
import json
import re
from decimal import Decimal
from pydantic import BaseModel
class MyModel(BaseModel):
value: Decimal | None
schema = MyModel.model_json_schema(mode="serialization")
pattern = next(
v for v in schema["properties"]["value"]["anyOf"] if v.get("type") == "string"
)["pattern"]
regex = re.compile(pattern)
# pydantic serializes this as "1E-7" which fails its own schema's pattern
m = MyModel(value=Decimal("0.0000001"))
serialized = json.loads(m.model_dump_json())["value"]
print(f"Pattern: {pattern}")
print(f"Serialized: {serialized!r}")
print(f"Matches: {bool(regex.match(serialized))}")
# Pattern: ^(?!^[-+.]*$)[+-]?0*\d*\.?\d*$
# Serialized: '1E-7'
# Matches: False
Python, Pydantic & OS Version
pydantic version: 2.12.5
pydantic-core version: 2.41.5
pydantic-core build: profile=release pgo=false
python version: 3.12.11 | packaged by conda-forge | (main, Jun 4 2025, 14:38:53) [Clang 18.1.8 ]
platform: macOS-26.2-arm64-arm-64bit
related packages: pyright-1.1.408 pydantic-extra-types-2.10.5 typing_extensions-4.15.0 fastapi-0.123.0 pydantic-settings-2.10.1
commit: unknownThe two patches
docs/concepts/json_schema.mdindex cd8bc01a..30a4551f 100644@@ -278,7 +278,7 @@ print(Model.model_json_schema(mode='validation')) 'anyOf': [ {'type': 'number'}, {- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', 'type': 'string', }, ],@@ -297,7 +297,7 @@ print(Model.model_json_schema(mode='serialization')) 'properties': { 'a': { 'default': '12.34',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', 'title': 'A', 'type': 'string', }pydantic/json_schema.pyindex 8135c60b..6b6feab3 100644@@ -761,7 +761,7 @@ class GenerateJsonSchema: # Case 4: Both are None (no restrictions) else:- pattern += r'\d*\.?\d*$' # look for arbitrary integer or decimal+ pattern += r'(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?$' # arbitrary integer or decimal, with optional exponent return pattern tests/test_json_schema.pyindex b1907b72..94eeaada 100644@@ -564,7 +564,7 @@ def test_decimal_json_schema(): {'type': 'number'}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ], 'default': '12.34',@@ -581,7 +581,7 @@ def test_decimal_json_schema(): 'default': '12.34', 'title': 'B', 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, }, 'title': 'Model',@@ -1172,7 +1172,7 @@ def test_special_decimal_types(field_type, expected_schema): {'type': 'number'}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ], 'title': 'A',@@ -2175,7 +2175,7 @@ def test_docstring(docstring, description): {'exclusiveMinimum': 2.0, 'type': 'number'}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ] },@@ -2188,7 +2188,7 @@ def test_docstring(docstring, description): {'type': 'number', 'exclusiveMaximum': 5}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ] },@@ -2201,7 +2201,7 @@ def test_docstring(docstring, description): {'type': 'number', 'minimum': 2}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ] },@@ -2214,7 +2214,7 @@ def test_docstring(docstring, description): {'type': 'number', 'maximum': 5}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ] },@@ -2227,7 +2227,7 @@ def test_docstring(docstring, description): {'type': 'number', 'multipleOf': 5}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ] },@@ -2276,7 +2276,7 @@ def test_constraints_schema_validation(kwargs, type_, expected_extra): Decimal, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ), (@@ -2284,7 +2284,7 @@ def test_constraints_schema_validation(kwargs, type_, expected_extra): Decimal, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ), (@@ -2292,7 +2292,7 @@ def test_constraints_schema_validation(kwargs, type_, expected_extra): Decimal, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ), (@@ -2300,7 +2300,7 @@ def test_constraints_schema_validation(kwargs, type_, expected_extra): Decimal, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ), (@@ -2308,7 +2308,7 @@ def test_constraints_schema_validation(kwargs, type_, expected_extra): Decimal, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ), ],@@ -6185,14 +6185,14 @@ def test_generate_definitions_for_no_ref_schemas(): { ('Decimal', 'serialization'): { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ('Decimal', 'validation'): { 'anyOf': [ {'type': 'number'}, { 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',+ 'pattern': '^(?!^[-+.]*$)[+-]?0*(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?$', }, ] },@@ -7391,6 +7391,38 @@ def test_decimal_pattern_reject_invalid_not_numerical_values_with_decimal_places assert re.fullmatch(pattern, invalid_decimal) is None +@pytest.mark.parametrize(+ 'value,is_scientific',+ [+ (Decimal('0.0000001'), True),+ (Decimal('1E+30'), True),+ (Decimal('-1.5E-10'), True),+ (Decimal('12.34'), False),+ (Decimal('-0.5'), False),+ (Decimal('100'), False),+ ],+)+def test_decimal_pattern_matches_serialized_scientific_notation(value: Decimal, is_scientific: bool) -> None:+ """Decimals serialized in scientific notation (e.g. `1E-7`) must match the JSON Schema pattern."""++ class Model(BaseModel):+ value: Decimal | None++ schema = Model.model_json_schema(mode='serialization')+ string_branch = next(s for s in schema['properties']['value']['anyOf'] if s.get('type') == 'string')+ pattern = string_branch['pattern']++ serialized = json.loads(Model(value=value).model_dump_json())['value']+ assert ('E' in serialized) is is_scientific+ assert re.fullmatch(pattern, serialized) is not None+++@pytest.mark.parametrize('invalid_decimal', ['', '-', '.', 'e5', 'E', '1e', '1e+', 'abc', '1.2.3'])+def test_decimal_pattern_reject_invalid_scientific_notation(invalid_decimal, get_decimal_pattern) -> None:+ pattern = get_decimal_pattern()+ assert re.fullmatch(pattern, invalid_decimal) is None++ def test_union_format_primitive_type_array() -> None: class Sub(BaseModel): pass
docs/api/standard_library_types.mdindex 4070fda59..5802bb2e3 100644@@ -300,11 +300,48 @@ Decimals support the following constraints (numbers must be coercible to decimal | `gt` | The value must be strictly greater than this number | [`exclusiveMinimum`](https://json-schema.org/understanding-json-schema/reference/numeric#range) keyword | | `multiple_of` | The value must be a multiple of this number | [`multipleOf`](https://json-schema.org/understanding-json-schema/reference/numeric#multiples) keyword | | `allow_inf_nan` | Whether to allow NaN (not-a-number) and infinite values | N/A |-| `max_digits` | The maximum number of decimal digits allowed. The zero before the decimal point and trailing zeros are not counted. | [`pattern`](https://json-schema.org/understanding-json-schema/reference/string#regexp) keyword, to describe the string pattern |-| `decimal_places` | The maximum number of decimal places allowed. Trailing zeros are not counted. | [`pattern`](https://json-schema.org/understanding-json-schema/reference/string#regexp) keyword, to describe the string pattern |+| `max_digits` | The maximum number of decimal digits allowed. The zero before the decimal point and trailing zeros are not counted. | N/A (see below) |+| `decimal_places` | The maximum number of decimal places allowed. Trailing zeros are not counted. | N/A (see below) | -Note that the JSON Schema [`pattern`](https://json-schema.org/understanding-json-schema/reference/string#regexp) keyword will be specified-in the JSON Schema to describe the string pattern in all cases (and can vary if `max_digits` and/or `decimal_places` is specified).+/// version-changed | v2.12+The JSON Schema includes a [`pattern`](https://json-schema.org/understanding-json-schema/reference/string#regexp) matching the `max_digits`+and `decimal_places` constraints.+///++/// version-changed | v2.14+The JSON Schema no longer includes a [`pattern`](https://json-schema.org/understanding-json-schema/reference/string#regexp) by default,+as the generated pattern can cause issues for downstream consumers due to its complexity.++The pattern can be included by subclassing [`GenerateJsonSchema`][pydantic.json_schema.GenerateJsonSchema] and overriding the+[`get_decimal_pattern()`][pydantic.json_schema.GenerateJsonSchema.get_decimal_pattern] method:++```python+from decimal import Decimal+from typing import Annotated++from pydantic import Field, TypeAdapter+from pydantic.json_schema import GenerateJsonSchema+++class MyGenerateJsonSchema(GenerateJsonSchema):+ def get_decimal_pattern(self, schema):+ return self.build_decimal_pattern(schema)+++ta = TypeAdapter(Annotated[Decimal, Field(max_digits=5, decimal_places=2)])+print(+ ta.json_schema(schema_generator=MyGenerateJsonSchema, mode='serialization')+)+"""+{+ 'pattern': '^-?(?:(?:0|[1-9][0-9]{0,2})(?:\\.[0-9]{1,2}0*)?|0E[+-][1-9][0-9]*|[1-9](?:\\.[0-9]+)?E\\+[1-2])$',+ 'type': 'string',+}+"""+```++The generated pattern also no longer uses lookahead assertions, and takes exponents and `allow_inf_nan` into account.+/// These constraints can be provided using the [`Field()`][pydantic.Field] function. The `Le`, `Ge`, `Lt`, `Gt` and `MultipleOf` metadata types from the [`annotated-types`](https://github.com/annotated-types/annotated-types)docs/concepts/json_schema.mdindex cd8bc01a8..e0a7bdf4c 100644@@ -275,13 +275,7 @@ print(Model.model_json_schema(mode='validation')) { 'properties': { 'a': {- 'anyOf': [- {'type': 'number'},- {- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- 'type': 'string',- },- ],+ 'anyOf': [{'type': 'number'}, {'type': 'string'}], 'default': '12.34', 'title': 'A', }@@ -294,14 +288,7 @@ print(Model.model_json_schema(mode='validation')) print(Model.model_json_schema(mode='serialization')) """ {- 'properties': {- 'a': {- 'default': '12.34',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- 'title': 'A',- 'type': 'string',- }- },+ 'properties': {'a': {'default': '12.34', 'title': 'A', 'type': 'string'}}, 'title': 'Model', 'type': 'object', }pydantic/json_schema.pyindex 8135c60b5..5c585426d 100644@@ -723,49 +723,11 @@ class GenerateJsonSchema: Returns: The generated JSON schema. """-- def get_decimal_pattern(schema: core_schema.DecimalSchema) -> str:- max_digits = schema.get('max_digits')- decimal_places = schema.get('decimal_places')-- pattern = (- r'^(?!^[-+.]*$)[+-]?0*' # check it is not empty string and not one or sequence of ".+-" characters.- )-- # Case 1: Both max_digits and decimal_places are set- if max_digits is not None and decimal_places is not None:- integer_places = max(0, max_digits - decimal_places)- pattern += (- rf'(?:'- rf'\d{{0,{integer_places}}}'- rf'|'- rf'(?=[\d.]{{1,{max_digits + 1}}}0*$)'- rf'\d{{0,{integer_places}}}\.\d{{0,{decimal_places}}}0*$'- rf')'- )-- # Case 2: Only max_digits is set- elif max_digits is not None and decimal_places is None:- pattern += (- rf'(?:'- rf'\d{{0,{max_digits}}}'- rf'|'- rf'(?=[\d.]{{1,{max_digits + 1}}}0*$)'- rf'\d*\.\d*0*$'- rf')'- )-- # Case 3: Only decimal_places is set- elif max_digits is None and decimal_places is not None:- pattern += rf'\d*\.?\d{{0,{decimal_places}}}0*$'-- # Case 4: Both are None (no restrictions)- else:- pattern += r'\d*\.?\d*$' # look for arbitrary integer or decimal-- return pattern-- json_schema = self.str_schema(core_schema.str_schema(pattern=get_decimal_pattern(schema)))+ str_schema = core_schema.str_schema()+ pattern = self.get_decimal_pattern(schema)+ if pattern is not None:+ str_schema['pattern'] = pattern+ json_schema = self.str_schema(str_schema) if self.mode == 'validation': multiple_of = schema.get('multiple_of') le = schema.get('le')@@ -789,6 +751,63 @@ class GenerateJsonSchema: } return json_schema + def get_decimal_pattern(self, schema: core_schema.DecimalSchema) -> str | None:+ """Get the regular expression pattern to apply to the string representation of a decimal.++ By default, no pattern is applied (`None` is returned). Subclasses can override this method+ to return a custom pattern, or [`build_decimal_pattern()`][pydantic.json_schema.GenerateJsonSchema.build_decimal_pattern]+ can be used to get a pattern reflecting the `max_digits` and `decimal_places` constraints:++ ```python+ from pydantic.json_schema import GenerateJsonSchema++ class MyGenerateJsonSchema(GenerateJsonSchema):+ def get_decimal_pattern(self, schema):+ return self.build_decimal_pattern(schema)+ ```++ Args:+ schema: The core schema.++ Returns:+ The regular expression pattern, or `None` if no pattern should be applied.+ """+ return None++ def build_decimal_pattern(self, schema: core_schema.DecimalSchema) -> str:+ """Build a regular expression pattern for the string representation of a decimal.++ The pattern takes into account the `max_digits` and `decimal_places` constraints, and the current+ mode:++ - In `'validation'` mode, the pattern accepts the string syntax understood by the [`Decimal`][decimal.Decimal]+ constructor. The `max_digits` and `decimal_places` constraints are only reflected for strings+ *without* an exponent: strings using an exponent are accepted as long as they are syntactically valid,+ as the constraints can't be expressed with a regular expression in this case.+ - In `'serialization'` mode, the pattern exactly describes the strings produced by pydantic+ (i.e. by `str(decimal)`) for decimals satisfying the constraints,+ including the ones using scientific notation (e.g. `'1E-7'`, `'1.5E+3'`).++ In both modes, trailing zeros in the decimal part are not counted (e.g. `'1.10'` is accepted with+ `decimal_places=1`), consistent with the validation behavior. If `allow_inf_nan` is enabled (on the+ core schema or in the configuration), the string representations of infinity and NaN are also accepted.++ Args:+ schema: The core schema.++ Returns:+ The regular expression pattern.+ """+ max_digits = schema.get('max_digits')+ decimal_places = schema.get('decimal_places')+ allow_inf_nan = schema.get('allow_inf_nan')+ if allow_inf_nan is None:+ allow_inf_nan = self._config.config_dict.get('allow_inf_nan', False)+ if self.mode == 'validation':+ return _decimal_validation_pattern(max_digits, decimal_places, allow_inf_nan)+ else:+ return _decimal_serialization_pattern(max_digits, decimal_places, allow_inf_nan)+ def str_schema(self, schema: core_schema.StringSchema) -> JsonSchemaValue: """Generates a JSON schema that matches a string value. @@ -2773,15 +2792,153 @@ def models_json_schema( # ##### End JSON Schema Generation Functions ##### -_HashableJsonValue: TypeAlias = (- int | float | str | bool | None | tuple['_HashableJsonValue', ...] | tuple[tuple[str, '_HashableJsonValue'], ...]-)+# Decimal JSON Schema pattern utilities.+# Note: while we could use `\d` to represent digits (`Decimal` objects allow Unicode decimal digits), we use [0-9] instead:+# - to reduce complexity (`\d` expands to a large set, and with `max_digits`/`decimal_places` constraints set, it can blow+# the compile limit on some engines).+# - [0-9] is a portable pattern (e.g. `\d` is equivalent to `[0-9]` in ECMA (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Character_class_escape#d)+# while it allows any Unicode decimal digit in Python.+++def _repeat(atom: str, min_count: int, max_count: int) -> str:+ """Build a regex repetition of `atom` between `min_count` and `max_count` times (inclusive), using the shortest quantifier."""+ if min_count > max_count or max_count == 0:+ return ''+ if min_count == max_count:+ return atom if min_count == 1 else f'{atom}{{{min_count}}}'+ if (min_count, max_count) == (0, 1):+ return f'{atom}?'+ return f'{atom}{{{min_count},{max_count}}}'+++def _int_range_pattern(max_value: int) -> str:+ """Build a regex matching the integers between 1 and `max_value` (inclusive), without leading zeros."""+ digits = str(max_value)+ length = len(digits)+ alts: list[str] = [f'[1-9]{_repeat("[0-9]", k - 1, k - 1)}' for k in range(1, length)]+ for i, char in enumerate(digits):+ d = int(char)+ low = 1 if i == 0 else 0+ high = d if i == length - 1 else d - 1+ if high >= low:+ char_class = str(low) if low == high else f'[{low}-{high}]'+ alts.append(f'{digits[:i]}{char_class}{_repeat("[0-9]", length - i - 1, length - i - 1)}')+ return alts[0] if len(alts) == 1 else f'(?:{"|".join(alts)})'+++def _decimal_validation_pattern(max_digits: int | None, decimal_places: int | None, allow_inf_nan: bool) -> str:+ """Build the decimal pattern for the `'validation'` mode."""++ number = r'(?:[0-9]+\.?[0-9]*|\.[0-9]+)' # e.g. 1, 1., .5+ exponent = r'[eE][+-]?[0-9]+' # e.g. e5, E5, e-7+ # e.g. Infinity, inf, NaN, sNaN, NaN123 (case-insensitive). Spelled out explicitly, as inline flags+ # aren't supported by all regular expression engines:+ non_finite = r'|[iI][nN][fF](?:[iI][nN][iI][tT][yY])?|[sS]?[nN][aA][nN][0-9]*' if allow_inf_nan else ''+ if max_digits is None and decimal_places is None:+ return rf'^[+-]?(?:{number}(?:{exponent})?{non_finite})$'++ alts: list[str] = []+ if max_digits is not None and decimal_places is not None:+ # If `decimal_places > max_digits`, no whole digits are allowed and `max_digits` bounds the decimal places:+ whole = max(0, max_digits - decimal_places)+ places = min(max_digits, decimal_places)+ int_part = f'0*{_repeat("[0-9]", 1, whole)}' if whole else '0+'+ opt_int_part = f'0*{_repeat("[0-9]", 0, whole)}'+ opt_frac_part = rf'\.{_repeat("[0-9]", 0, places)}0*'+ frac_part = rf'\.{_repeat("[0-9]", 1, places)}0*' if places else r'\.0+'+ alts += [f'{int_part}(?:{opt_frac_part})?', f'{opt_int_part}{frac_part}']+ elif max_digits is not None:+ # The whole part and decimal part must have at most `max_digits` digits in total,+ # so we enumerate the possible widths of the whole part:+ int_part = f'0*{_repeat("[0-9]", 1, max_digits)}' if max_digits else '0+'+ alts.append(rf'{int_part}(?:\.0*)?')+ alts += [+ rf'0*{_repeat("[0-9]", whole, whole)}\.{_repeat("[0-9]", 1, max_digits - whole)}0*'+ for whole in range(max_digits)+ ] or [r'0*\.0+']+ else:+ assert decimal_places is not None+ opt_frac_part = rf'\.{_repeat("[0-9]", 0, decimal_places)}0*'+ frac_part = rf'\.{_repeat("[0-9]", 1, decimal_places)}0*' if decimal_places else r'\.0+'+ alts += [f'[0-9]+(?:{opt_frac_part})?', f'[0-9]*{frac_part}']++ alts.append(f'{number}{exponent}')+ return rf'^[+-]?(?:{"|".join(alts)}{non_finite})$'+++def _decimal_serialization_pattern(max_digits: int | None, decimal_places: int | None, allow_inf_nan: bool) -> str:+ """Build the decimal pattern for the `'serialization'` mode.++ Decimals are serialized using `str()`, producing either:++ - The plain notation (`'123.45'`), when the exponent is lower than or equal to 0 and the adjusted+ exponent is greater than or equal to -6.+ - The scientific notation otherwise (`'1E+3'`, `'1.5E-7'`, `'0E-10'`), with a single digit before+ the optional decimal point and an always signed exponent.++ Trailing zeros of the decimal part are preserved by `str()`, but aren't counted by the validation logic.+ """++ def frac_part(places: int) -> str:+ return rf'\.{_repeat("[0-9]", 1, places)}0*' if places else r'\.0+'++ alts: list[str] = []++ # Plain notation:+ if max_digits is not None and decimal_places is not None:+ whole = max(0, max_digits - decimal_places)+ places = min(max_digits, decimal_places)+ int_part = f'(?:0|[1-9]{_repeat("[0-9]", 0, whole - 1)})' if whole else '0'+ alts.append(f'{int_part}(?:{frac_part(places)})?')+ elif max_digits is not None:+ # The whole part and decimal part must have at most `max_digits` digits in total,+ # so we enumerate the possible widths of the whole part:+ alts.append(rf'0(?:\.{_repeat("[0-9]", 1, max_digits)}0*)?')+ alts += [+ f'[1-9]{_repeat("[0-9]", whole - 1, whole - 1)}(?:{frac_part(max_digits - whole)})?'+ for whole in range(1, max_digits + 1)+ ]+ elif decimal_places is not None:+ alts.append(f'(?:0|[1-9][0-9]*)(?:{frac_part(decimal_places)})?')+ else:+ alts.append(r'(?:0|[1-9][0-9]*)(?:\.[0-9]+)?')++ # Zero with an exponent (e.g. `'0E-10'`), which normalizes to zero and thus satisfies any constraint:+ alts.append(r'0E[+-][1-9][0-9]*')++ # Scientific notation with a positive exponent `k` (e.g. `'1.5E+3'`): the value has `k + 1` whole digits+ # and no decimal places:+ if max_digits is None:+ alts.append(r'[1-9](?:\.[0-9]+)?E\+[1-9][0-9]*')+ else:+ max_exponent = max_digits - (decimal_places or 0) - 1+ if max_exponent >= 1:+ alts.append(rf'[1-9](?:\.[0-9]+)?E\+{_int_range_pattern(max_exponent)}')++ # Scientific notation with a negative exponent `k` (`k >= 7`, e.g. `'1.5E-7'`): the value has no whole digits,+ # and `k` + (the number of significant decimal digits of the coefficient) decimal places:+ if max_digits is None and decimal_places is None:+ alts.append(r'[1-9](?:\.[0-9]+)?E-[1-9][0-9]*')+ else:+ max_places = min(c for c in (max_digits, decimal_places) if c is not None)+ alts += [f'[1-9](?:{frac_part(max_places - k)})?E-{k}' for k in range(7, max_places + 1)]++ if allow_inf_nan:+ # `str()` produces `'Infinity'`, `'NaN'` and `'sNaN'` (with an optional payload):+ alts += ['Infinity', 's?NaN[0-9]*']++ return rf'^-?(?:{"|".join(alts)})$' def _deduplicate_schemas(schemas: Iterable[JsonDict]) -> list[JsonDict]: return list({_make_json_hashable(schema): schema for schema in schemas}.values()) +_HashableJsonValue: TypeAlias = (+ int | float | str | bool | None | tuple['_HashableJsonValue', ...] | tuple[tuple[str, '_HashableJsonValue'], ...]+)++ def _make_json_hashable(value: JsonValue) -> _HashableJsonValue: if isinstance(value, dict): return tuple(sorted((k, _make_json_hashable(v)) for k, v in value.items()))tests/test_json_schema.pyindex b1907b720..bb38003f8 100644@@ -36,6 +36,7 @@ from typing_extensions import TypeAliasType, TypedDict, deprecated import pydantic from pydantic import ( AfterValidator,+ AllowInfNan, BaseModel, BeforeValidator, Field,@@ -64,6 +65,7 @@ from pydantic.json_schema import ( DEFAULT_REF_TEMPLATE, Examples, GenerateJsonSchema,+ JsonSchemaMode, JsonSchemaValue, NoDefault, PydanticJsonSchemaWarning,@@ -562,10 +564,7 @@ def test_decimal_json_schema(): 'b': { 'anyOf': [ {'type': 'number'},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ], 'default': '12.34', 'title': 'B',@@ -581,7 +580,6 @@ def test_decimal_json_schema(): 'default': '12.34', 'title': 'B', 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$', }, }, 'title': 'Model',@@ -1170,10 +1168,7 @@ def test_special_decimal_types(field_type, expected_schema): 'a': { 'anyOf': [ {'type': 'number'},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ], 'title': 'A', }@@ -2173,10 +2168,7 @@ def test_docstring(docstring, description): { 'anyOf': [ {'exclusiveMinimum': 2.0, 'type': 'number'},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ] }, ),@@ -2186,10 +2178,7 @@ def test_docstring(docstring, description): { 'anyOf': [ {'type': 'number', 'exclusiveMaximum': 5},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ] }, ),@@ -2199,10 +2188,7 @@ def test_docstring(docstring, description): { 'anyOf': [ {'type': 'number', 'minimum': 2},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ] }, ),@@ -2212,10 +2198,7 @@ def test_docstring(docstring, description): { 'anyOf': [ {'type': 'number', 'maximum': 5},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ] }, ),@@ -2225,10 +2208,7 @@ def test_docstring(docstring, description): { 'anyOf': [ {'type': 'number', 'multipleOf': 5},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ] }, ),@@ -2274,42 +2254,27 @@ def test_constraints_schema_validation(kwargs, type_, expected_extra): ( {'gt': 2}, Decimal,- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ), ( {'lt': 5}, Decimal,- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ), ( {'ge': 2}, Decimal,- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ), ( {'le': 5}, Decimal,- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ), ( {'multiple_of': 5}, Decimal,- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ), ], )@@ -6183,17 +6148,11 @@ def test_generate_definitions_for_no_ref_schemas(): ) assert result == ( {- ('Decimal', 'serialization'): {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ ('Decimal', 'serialization'): {'type': 'string'}, ('Decimal', 'validation'): { 'anyOf': [ {'type': 'number'},- {- 'type': 'string',- 'pattern': '^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$',- },+ {'type': 'string'}, ] }, ('Model', 'validation'): {'$ref': '#/$defs/Model'},@@ -7281,15 +7240,42 @@ def test_json_schema_arguments_v3_aliases() -> None: } +class DecimalPatternGenerateJsonSchema(GenerateJsonSchema):+ def get_decimal_pattern(self, schema: core_schema.DecimalSchema) -> str | None:+ return self.build_decimal_pattern(schema)++ @pytest.fixture def get_decimal_pattern():- def pattern(max_digits=None, decimal_places=None) -> str:- filed = TypeAdapter(Annotated[Decimal, Field(max_digits=max_digits, decimal_places=decimal_places)])- return filed.json_schema()['anyOf'][1]['pattern']+ def pattern(max_digits=None, decimal_places=None, mode: JsonSchemaMode = 'validation') -> str:+ ta = TypeAdapter(Annotated[Decimal, Field(max_digits=max_digits, decimal_places=decimal_places)])+ json_schema = ta.json_schema(schema_generator=DecimalPatternGenerateJsonSchema, mode=mode)+ if mode == 'validation':+ json_schema = json_schema['anyOf'][1]+ return json_schema['pattern'] return pattern +def test_decimal_pattern_not_set_by_default() -> None:+ ta = TypeAdapter(Annotated[Decimal, Field(max_digits=4, decimal_places=2)])++ assert ta.json_schema() == {'anyOf': [{'type': 'number'}, {'type': 'string'}]}+ assert ta.json_schema(mode='serialization') == {'type': 'string'}+++def test_decimal_pattern_custom() -> None:+ class CustomGenerateJsonSchema(GenerateJsonSchema):+ def get_decimal_pattern(self, schema: core_schema.DecimalSchema) -> str | None:+ return rf'^\d{{1,{schema["max_digits"]}}}$'++ ta = TypeAdapter(Annotated[Decimal, Field(max_digits=4)])++ assert ta.json_schema(schema_generator=CustomGenerateJsonSchema) == {+ 'anyOf': [{'type': 'number'}, {'type': 'string', 'pattern': r'^\d{1,4}$'}]+ }++ @pytest.mark.parametrize('valid_decimal', ['0.1', '0000.1', '11.1', '001.1', '11111111.1', '0.100000', '0.01', '0.11']) def test_decimal_pattern_with_only_decimal_places_set(valid_decimal, get_decimal_pattern) -> None: decimal_places = 2@@ -7391,6 +7377,162 @@ def test_decimal_pattern_reject_invalid_not_numerical_values_with_decimal_places assert re.fullmatch(pattern, invalid_decimal) is None +@pytest.mark.parametrize(['max_digits', 'decimal_places'], [(None, None), (3, None), (None, 2), (4, 2)])+@pytest.mark.parametrize(+ 'valid_decimal', ['1e5', '1E5', '1E+5', '1e-5', '-1.5E1', '+.5e1', '1.e5', '01.10E01', '1.234E+1', '12345E-3']+)+def test_decimal_pattern_with_exponent(max_digits, decimal_places, valid_decimal, get_decimal_pattern) -> None:+ pattern = get_decimal_pattern(max_digits, decimal_places)++ assert re.fullmatch(pattern, valid_decimal) is not None+++@pytest.mark.parametrize(['max_digits', 'decimal_places'], [(None, None), (3, None), (None, 2), (4, 2)])+@pytest.mark.parametrize('invalid_decimal', ['e5', '.e5', '1e', '1e+', '1E5.0', '1.5.E5', '1e5e5', '1e1.5', '1f5'])+def test_decimal_pattern_reject_invalid_with_exponent(+ max_digits, decimal_places, invalid_decimal, get_decimal_pattern+) -> None:+ pattern = get_decimal_pattern(max_digits, decimal_places)++ assert re.fullmatch(pattern, invalid_decimal) is None+++@pytest.mark.parametrize(+ ['max_digits', 'decimal_places', 'valid_decimals'],+ [+ (+ None,+ None,+ [+ '0',+ '-0',+ '-0.0',+ '0.00',+ '12.34',+ '100',+ '0.000001',+ '1E-7',+ '1.5E-7',+ '0E-10',+ '0E+5',+ '1E+2',+ '1.0E+3',+ '-1.5E-100',+ '1E+100',+ ],+ ),+ (3, None, ['0', '0.100', '100', '10.0', '1.23', '0.001', '0.001000', '1E+2', '0E-10', '0E+15']),+ (None, 2, ['0', '1.23', '1.230000', '123456789.12', '0.10', '1E+2', '1.5E+3', '0E-10', '1E+100']),+ (5, 2, ['0', '-0.10', '123.45', '123.450', '999', '99.9', '1E+2', '1.5E+2', '0E-10']),+ (2, 3, ['0.0', '0.01', '0.010', '-0.01', '0E-10']),+ (10, 8, ['0', '99.12345678', '99.1234567800', '0.00000001', '1E-7', '1.5E-7', '1E-8', '1E+1', '0E-20']),+ (30, None, ['1E-7', '1.2345678901234567890123E-7', '1E-30', '1E+29', '123456789012345678901234567890']),+ (4, 0, ['0', '9999', '9999.0', '1E+3', '1.5E+3', '0E-10']),+ ],+)+def test_decimal_pattern_serialization(max_digits, decimal_places, valid_decimals, get_decimal_pattern) -> None:+ pattern = get_decimal_pattern(max_digits, decimal_places, mode='serialization')+ ta = TypeAdapter(Annotated[Decimal, Field(max_digits=max_digits, decimal_places=decimal_places)])++ for valid_decimal in valid_decimals:+ serialized = json.loads(ta.dump_json(ta.validate_python(valid_decimal)))+ assert serialized == str(Decimal(valid_decimal))+ assert re.fullmatch(pattern, serialized) is not None, valid_decimal+++@pytest.mark.parametrize(+ ['max_digits', 'decimal_places', 'invalid_decimals'],+ [+ # Not produced by `str()` (or not a decimal):+ (+ None,+ None,+ [+ '',+ '.',+ '-',+ '+1',+ '01',+ '1.',+ '.1',+ '-01.5',+ '1e5',+ '1E5',+ '1E-05',+ '1E+0',+ '0E-0',+ '00E-10',+ '10E+2',+ '0.0E+2',+ 'E5',+ 'Infinity',+ 'NaN',+ ],+ ),+ # Doesn't satisfy the constraints:+ (3, None, ['1000', '1.234', '0.0001', '1E+3', '1E-4', '1.5E+3', '1.5E-7']),+ (None, 2, ['1.234', '0.001', '1E-7', '1.5E-7', '0.001E+2']),+ (5, 2, ['1234', '123.456', '1000.5', '0.001', '1E+3', '1E-7']),+ (2, 3, ['1', '0.001', '1.0', '1E+1', '1E-7']),+ (10, 8, ['100', '99.123456789', '0.000000001', '1E-9', '1.5E-8', '1.23E-7', '1E+2']),+ (4, 0, ['0.1', '10000', '1E+4', '1E-7']),+ ],+)+def test_decimal_pattern_serialization_reject_invalid(+ max_digits, decimal_places, invalid_decimals, get_decimal_pattern+) -> None:+ pattern = get_decimal_pattern(max_digits, decimal_places, mode='serialization')++ for invalid_decimal in invalid_decimals:+ assert re.fullmatch(pattern, invalid_decimal) is None, invalid_decimal+++def test_decimal_pattern_allow_inf_nan() -> None:+ class Model(BaseModel):+ model_config = ConfigDict(allow_inf_nan=True)++ a: Decimal+ b: Annotated[Decimal, AllowInfNan(False)] = Decimal(0)++ class ModelNoConfig(BaseModel):+ a: Annotated[Decimal, AllowInfNan()]++ non_finite = ['Infinity', '-Infinity', 'inf', 'INF', '+Inf', 'NaN', 'nan', '-NaN', 'sNaN', 'snan', 'NaN123']++ for model, field in [(Model, 'a'), (ModelNoConfig, 'a')]:+ validation_pattern = model.model_json_schema(schema_generator=DecimalPatternGenerateJsonSchema)['properties'][+ field+ ]['anyOf'][1]['pattern']+ serialization_pattern = model.model_json_schema(+ schema_generator=DecimalPatternGenerateJsonSchema, mode='serialization'+ )['properties'][field]['pattern']++ for value in ['1.5', *non_finite]:+ assert re.fullmatch(validation_pattern, value) is not None, value+ serialized = model.model_validate({field: value}).model_dump(mode='json')[field]+ assert re.fullmatch(serialization_pattern, serialized) is not None, serialized++ for value in ['Infinity1', 'inf.0', 'in', 'NaN1.5', 'sInf', 'i n f']:+ assert re.fullmatch(validation_pattern, value) is None, value+ for value in ['inf', 'nan', 'INFINITY', '+Infinity', 'sNaN1.5', 'sInf']:+ assert re.fullmatch(serialization_pattern, value) is None, value++ # The schema takes precedence over the config:+ for mode in ('validation', 'serialization'):+ json_schema = Model.model_json_schema(schema_generator=DecimalPatternGenerateJsonSchema, mode=mode)+ pattern = json_schema['properties']['b']+ pattern = (pattern['anyOf'][1] if mode == 'validation' else pattern)['pattern']+ for value in non_finite:+ assert re.fullmatch(pattern, value) is None, value+++def test_decimal_pattern_allow_inf_nan_disabled_by_default(get_decimal_pattern) -> None:+ for mode in ('validation', 'serialization'):+ pattern = get_decimal_pattern(mode=mode)+ for value in ['Infinity', '-Infinity', 'inf', 'NaN', 'sNaN']:+ assert re.fullmatch(pattern, value) is None, value++ def test_union_format_primitive_type_array() -> None: class Sub(BaseModel): pass
What the agent reported
The Decimal JSON schema pattern now accepts scientific notation, so values like "1E-7" from model_dump_json() match the schema's own pattern. The JSON schema and docs tests pass.
What was wrong: pydantic-core serializes Decimal with str(), which switches to scientific notation for very small or large values (Decimal('0.0000001') becomes "1E-7"). The pattern for an unconstrained Decimal in get_decimal_pattern (pydantic/json_schema.py, around line 764) had no exponent part, so it rejected those strings.
What I changed:
- The fix: one line in that no-constraints case. The pattern is now
^(?!^[-+.]*$)[+-]?0*(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?$. Just appending(?:[eE][+-]?\d+)?to the old pattern would also have matched junk like"e5", so the number part now requires at least one digit. Every valid number the old pattern accepted still matches (0,00,1.,.5,-0.5,+3). - Both modes: the change applies to validation and serialization schemas, since they share this string and validation accepts
"1E-7"too. - Existing tests and docs: 15 places in
tests/test_json_schema.pyand 2 indocs/concepts/json_schema.mdchecked the exact old pattern text. I replaced only that string; nothing was skipped or loosened. - New tests in
tests/test_json_schema.py: test_decimal_pattern_matches_serialized_scientific_notationis the report's scenario: avalue: Decimal | Nonemodel, the pattern frommodel_json_schema(mode='serialization'), and a full match ofmodel_dump_json()output for1E-7,1E+30,-1.5E-10,12.34,-0.5and100. It also asserts the first three really serialize with anE.test_decimal_pattern_reject_invalid_scientific_notationchecks the new pattern still rejects'',-,.,e5,E,1e,1e+,abcand1.2.3.
Test evidence:
./bt-test tests/test_json_schema.py tests/test_docs.py tests/test_types.py tests/test_main.py: 1725 passed, 131 skipped, 3 xfailed, 0 failures.- After the reviewer's cleanups,
./bt-test tests/test_json_schema.py tests/test_docs.py: 1188 passed, 103 skipped, 1 xfailed. - Before-fix check: I didn't run the test file against the old code. Instead I ran Python's
re.fullmatchon the old and new patterns:1E-7,1E+30and-1.5E-10fail the old one and pass the new one, so the new test would have failed before the fix. ruffisn't installed here, so lint wasn't run. The reviewer's style points (an extra blank line,re.matchinstead of the file's usualre.fullmatch) are fixed.
Requirements:
- Serialization pattern rejects
"1E-7": fixed, covered by the regression test above. - Fix option 1, add an exponent to the pattern: done, in a slightly stricter form so
"e5"isn't accepted. - Fix option 2, serialize
Decimalwithout scientific notation: not done. That output comes from pydantic-core, not this repository, and changing it would alter the JSON every user currently gets. The pattern fix resolves the mismatch on its own. - The report's example (
Decimal | None, serialization mode,1E-7): covered by the regression test.
Still open (not in the report):
- Fields with
max_digitsordecimal_places: these use separate patterns I didn't change, and they still reject scientific-notation output (e.g.Decimal('1E+3')withdecimal_places=2serializes as"1E+3"). Counting digits across an exponent isn't practical in a regex. NaNandInfinity: these were already rejected by the old pattern and still are.
All changes are uncommitted in pydantic/json_schema.py, tests/test_json_schema.py and docs/concepts/json_schema.md.