Backtestspytestrun-20261106-234554#10210
pytest.approx does not raise in mixed data type nested structures
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the root cause the same way: they replace the `isinstance(x, type(expected))` check with a generic 'is this a non-string Collection' predicate in both ApproxMapping and ApproxSequenceLike, and every independent test passes on both. The differences are in craft and tests. Patch B distinguishes, inside a mapping, a nested Mapping (keeps the 'nested dictionaries' message) from other nested containers (uses the 'nested data structures' message), so `approx({"k": [1]})` reports accurately. Patch A routes every nested value in a dict through the 'nested dictionaries' message, so a list inside a dict is reported as a nested dictionary, which is misleading, and A's tests codify that inaccurate wording. Patch B's parametrized construction test is also broader (list/tuple/dict parents each paired with list/tuple/set/dict children) and its comparison test mirrors the issue's exact reproductions including the equal-values case. Patch A has one small advantage: it excludes bytearray as well as str/bytes, and adds tests for bytes leaves, whereas B would now raise for a bytearray element, an obscure edge the issue does not mention. B's extra `or _is_sequence_like(x)` is redundant but harmless. On balance B's more accurate error reporting and more thorough tests outweigh A's bytearray handling.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
4m 18s
Wall-clock time in the container
Full test suite
No new failures
725 passed · 4 skipped · 729 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
44vs 51 merged
Changed lines, ours vs merged
The issue
View on GitHub<!--
Thanks for submitting an issue!
Quick check-list while reporting bugs:
-->
description
when using pytest.approx on nested data structures I'd expect it to raise an exception also for mixed data type containers like a tuple in a list. So far it only works for the same data types e.g. lists in lists, tuples in tuples.
For mixed data structures it incorrectly returns False, even though they should match which is a little misleading (also see below).
output of pip list
(venv) kittnjdr@ububox:~/workspace/pytest$ pip list
Package Version Editable project location
---------- ----------------------- -------------------------------
attrs 22.1.0
iniconfig 1.1.1
packaging 21.3
pip 22.1.2
pluggy 1.0.0
py 1.11.0
pyparsing 3.0.9
pytest 7.2.0.dev232+gcc0092b9d /home/kittnjdr/workspace/pytest
setuptools 63.2.0
tomli 2.0.1
wheel 0.37.1
versions
- pytest: main branch
- OS: ubuntu 22.04
minimal reproduction
>>> import pytest
>>> [(1.0001,)] == pytest.approx([(1,)], rel=1e-2)
False
>>> ({1.0001},) == pytest.approx(({1},), rel=1e-2)
False
>>> ({"key": 1.0001},) == pytest.approx(({"key": 1},), rel=1e-2)
False
>>> (1.0001,) == pytest.approx((1,), rel=1e-2)
True
>>> ((1.0001,),) == pytest.approx(((1,),), rel=1e-2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/kittnjdr/workspace/pytest/src/_pytest/python_api.py", line 744, in approx
return cls(expected, rel, abs, nan_ok)
File "/home/kittnjdr/workspace/pytest/src/_pytest/python_api.py", line 86, in __init__
self._check_type()
File "/home/kittnjdr/workspace/pytest/src/_pytest/python_api.py", line 381, in _check_type
raise TypeError(msg.format(x, index, pprint.pformat(self.expected)))
TypeError: pytest.approx() does not support nested data structures: (1,) at index 0
full sequence: ((1,),)
however, it works fine if both are exactly the same, but here I'd also expect an exception instead of a result (which is correct, but maybe unintentional)
>>> [(1.0,)] == pytest.approx([(1.0,)])
True
test cases
These test cases fail, but should pass with the potential bug fixed.
diff --git a/testing/python/approx.py b/testing/python/approx.py
index 6acb466ff..31dd4638b 100644
--- a/testing/python/approx.py
+++ b/testing/python/approx.py
@@ -781,6 +781,10 @@ class TestApprox:
"x, name",
[
pytest.param([[1]], "data structures", id="nested-list"),
+ pytest.param([(1,)], "data structures", id="nested-list-tuple"),
+ pytest.param([{1}], "data structures", id="nested-list-set"),
+ pytest.param([{"key": 1}], "data structures", id="nested-list-dict"),
pytest.param({"key": {"key": 1}}, "dictionaries", id="nested-dict"),
],
)The two patches
changelog/10194.bugfix.rstnew file mode 100644index 000000000..5a3e46ab8@@ -0,0 +1 @@+:func:`pytest.approx` now raises :class:`TypeError` when the expected value contains a nested data structure of a different type than its container (for example a tuple inside a list, or a list inside a dictionary), instead of silently returning an incorrect comparison result.src/_pytest/approx.pyindex d040c249e..52f4e8f15 100644@@ -260,9 +260,12 @@ class ApproxMapping(Approx[Mapping[Any, Any]]): __tracebackhide__ = True for key, value in expected.items():- if isinstance(value, type(expected)):+ if isinstance(value, Mapping): msg = "pytest.approx() does not support nested dictionaries: key={!r} value={!r}\n full mapping={}" raise TypeError(msg.format(key, value, pprint.pformat(expected)))+ if _is_nested_data_structure(value):+ msg = "pytest.approx() does not support nested data structures: key={!r} value={!r}\n full mapping={}"+ raise TypeError(msg.format(key, value, pprint.pformat(expected))) super().__init__(expected, rel=rel, abs=abs, nan_ok=nan_ok) @@ -358,7 +361,7 @@ class ApproxSequenceLike(Approx[Sequence[Any]]): __tracebackhide__ = True for index, x in enumerate(expected):- if isinstance(x, type(expected)):+ if _is_nested_data_structure(x): msg = "pytest.approx() does not support nested data structures: {!r} at index {}\n full sequence: {}" raise TypeError(msg.format(x, index, pprint.pformat(expected))) @@ -979,6 +982,14 @@ def _is_sequence_like(expected: object) -> TypeGuard[Sequence[Any]]: ) +def _is_nested_data_structure(x: object) -> bool:+ """Return True if ``x`` is a container that cannot be an element of an+ expected sequence or mapping (strings and bytes are treated as scalars)."""+ if isinstance(x, str | bytes):+ return False+ return isinstance(x, Collection) or _is_sequence_like(x)++ def _as_numpy_array(obj: object) -> ndarray | None: """ Return an ndarray if the given object is implicitly convertible to ndarray,testing/python/approx.pyindex 8cac6f454..ce3c7522d 100644@@ -923,7 +923,17 @@ class TestApprox: "x, name", [ pytest.param([[1]], "data structures", id="nested-list"),+ pytest.param([(1,)], "data structures", id="nested-list-tuple"),+ pytest.param([{1}], "data structures", id="nested-list-set"),+ pytest.param([{"key": 1}], "data structures", id="nested-list-dict"),+ pytest.param(((1,),), "data structures", id="nested-tuple"),+ pytest.param(([1],), "data structures", id="nested-tuple-list"),+ pytest.param(({1},), "data structures", id="nested-tuple-set"),+ pytest.param(({"key": 1},), "data structures", id="nested-tuple-dict"), pytest.param({"key": {"key": 1}}, "dictionaries", id="nested-dict"),+ pytest.param({"key": [1]}, "data structures", id="nested-dict-list"),+ pytest.param({"key": (1,)}, "data structures", id="nested-dict-tuple"),+ pytest.param({"key": {1}}, "data structures", id="nested-dict-set"), ], ) def test_expected_value_type_error(self, x, name):@@ -933,6 +943,24 @@ class TestApprox: ): approx(x) + @pytest.mark.parametrize(+ "actual, expected",+ [+ pytest.param([(1.0001,)], [(1,)], id="list-tuple"),+ pytest.param(({1.0001},), ({1},), id="tuple-set"),+ pytest.param(({"key": 1.0001},), ({"key": 1},), id="tuple-dict"),+ pytest.param([(1.0,)], [(1.0,)], id="list-tuple-equal"),+ ],+ )+ def test_mixed_nested_data_structures_raise(self, actual, expected):+ """Comparing against mixed nested containers raises instead of+ silently returning a wrong result (#10194)."""+ with pytest.raises(+ TypeError,+ match=r"pytest.approx\(\) does not support nested data structures:",+ ):+ assert actual == approx(expected, rel=1e-2)+ @pytest.mark.parametrize( "x", [
AUTHORSindex 2805077f4..d1a2d3e79 100644@@ -373,6 +373,7 @@ Omar Kohl Omer Hadari Omri Golan Ondřej Súkup+Onkesh Bansal Oscar Benjamin Parth Patel Patrick Hayeschangelog/10210.bugfix.rstnew file mode 100644index 000000000..d89cbb02c@@ -0,0 +1,5 @@+:func:`pytest.approx` now raises a ``TypeError`` for any nested container, not+only for one of the same type as its parent. Previously ``approx([{"k": 1}])``+and ``approx({"k": [1]})`` were accepted and then compared unequal to every+value, silently ignoring the tolerance, while ``approx([[1]])`` correctly+reported that nesting is unsupported.src/_pytest/approx.pyindex d040c249e..780de1467 100644@@ -260,7 +260,7 @@ class ApproxMapping(Approx[Mapping[Any, Any]]): __tracebackhide__ = True for key, value in expected.items():- if isinstance(value, type(expected)):+ if _is_nested_container(value): msg = "pytest.approx() does not support nested dictionaries: key={!r} value={!r}\n full mapping={}" raise TypeError(msg.format(key, value, pprint.pformat(expected))) @@ -358,7 +358,7 @@ class ApproxSequenceLike(Approx[Sequence[Any]]): __tracebackhide__ = True for index, x in enumerate(expected):- if isinstance(x, type(expected)):+ if _is_nested_container(x): msg = "pytest.approx() does not support nested data structures: {!r} at index {}\n full sequence: {}" raise TypeError(msg.format(x, index, pprint.pformat(expected))) @@ -979,6 +979,17 @@ def _is_sequence_like(expected: object) -> TypeGuard[Sequence[Any]]: ) +def _is_nested_container(value: object) -> bool:+ """Whether ``value`` is a container ``approx`` cannot descend into.++ ``str``, ``bytes`` and ``bytearray`` are collections as well, but they+ compare exactly and are handled as leaves, so they are not nesting.+ """+ return isinstance(value, Collection) and not isinstance(+ value, str | bytes | bytearray+ )++ def _as_numpy_array(obj: object) -> ndarray | None: """ Return an ndarray if the given object is implicitly convertible to ndarray,testing/python/approx.pyindex 8cac6f454..c63529791 100644@@ -923,7 +923,14 @@ class TestApprox: "x, name", [ pytest.param([[1]], "data structures", id="nested-list"),+ pytest.param([(1,)], "data structures", id="list-of-tuple"),+ pytest.param([{1}], "data structures", id="list-of-set"),+ pytest.param([{"key": 1}], "data structures", id="list-of-dict"),+ pytest.param(({"key": 1},), "data structures", id="tuple-of-dict"), pytest.param({"key": {"key": 1}}, "dictionaries", id="nested-dict"),+ pytest.param({"key": [1]}, "dictionaries", id="dict-of-list"),+ pytest.param({"key": (1,)}, "dictionaries", id="dict-of-tuple"),+ pytest.param({"key": {1}}, "dictionaries", id="dict-of-set"), ], ) def test_expected_value_type_error(self, x, name):@@ -933,6 +940,27 @@ class TestApprox: ): approx(x) + @pytest.mark.parametrize(+ "expected, actual",+ [+ pytest.param([{"key": 1.0}], [{"key": 1.0 + 1e-9}], id="list-of-dict"),+ pytest.param({"key": [1.0]}, {"key": [1.0 + 1e-9]}, id="dict-of-list"),+ pytest.param([(1.0,)], [(1.0 + 1e-9,)], id="list-of-tuple"),+ pytest.param(({1.0},), ({1.0 + 1e-9},), id="tuple-of-set"),+ ],+ )+ def test_mixed_nested_containers_raise_instead_of_comparing_unequal(+ self, expected, actual+ ):+ """A nested container of a different type used to slip past the check.++ It was then compared as a leaf, so these all returned ``False``+ despite being well inside the default tolerance, silently ignoring+ it rather than reporting that nesting is unsupported (#10210).+ """+ with pytest.raises(TypeError, match=r"pytest.approx\(\) does not support"):+ actual == approx(expected)+ @pytest.mark.parametrize( "x", [@@ -940,6 +968,8 @@ class TestApprox: pytest.param("string"), pytest.param(["string"], id="nested-str"), pytest.param({"key": "string"}, id="dict-with-string"),+ pytest.param([b"bytes"], id="nested-bytes"),+ pytest.param({"key": b"bytes"}, id="dict-with-bytes"), ], ) def test_nonnumeric_okay_if_equal(self, x):