Backtestspytestrun-20261106-234554#15284
pytest.approx: 9.2.0 broke compatibility with JAX arrays
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the reported NotImplementedError, and the independent tests pass identically for both. The difference is in root cause vs. symptom. Patch A wraps the `.flat` lookup in `try/except NotImplementedError`, which is keyed to JAX's specific failure mode: any other array-like whose `.flat` raises a different exception (or returns something non-iterable) would break again, and the fix leaves `_contains_decimal` inspecting a foreign object's `.flat` even though `approx()` itself never compares the object that way. Patch B removes that reliance entirely by routing array-likes through `_as_numpy_array`, the same dispatch `approx()` uses for comparison, so the Decimal check now inspects exactly the ndarray that will be compared. That also makes the warning actually fire for array-likes of Decimals, which B covers with a dedicated test, and it drops the confusing `getattr(expected, 'flat', expected)` trick in favour of plain iteration for ordinary sequences. B's regression test covers 0-d and 1-d plus both `rel` and `abs`; A's covers 0-d/1-d/2-d and in/out of tolerance, so test quality is comparable. The main cost of B is one extra `__array__` conversion at approx construction time (a device-to-host copy for JAX), which is a performance tradeoff rather than a correctness problem; A's test explicitly asserts a single conversion, showing it avoids that. On balance B's structural fix is more robust and consistent with the surrounding code, while A is a narrower special case, so B wins on correctness and craft despite the extra conversion.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
3m 21s
Wall-clock time in the container
Full test suite
No new failures
761 passed · 4 skipped · 765 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
50vs 42 merged
Changed lines, ours vs merged
The issue
View on GitHub<!--
Thanks for submitting an issue!
Quick check-list while reporting bugs:
-->
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible
Description
#15006 broke compatibility with JAX arrays when used with pytest.approx.
Minimal example
import jax.numpy as jnp
import pytest
def test_jax_approx():
assert jnp.array(42) == pytest.approx(jnp.array(42), rel=1e-3)
With pytest 9.1.1: OK
With pytest 9.2.0:
========================================================== FAILURES ===========================================================
_______________________________________________________ test_jax_approx _______________________________________________________
def test_jax_approx():
> assert jnp.array(42) == pytest.approx(jnp.array(42), rel=1e-3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
test.py:5:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.14/site-packages/_pytest/approx.py:1062: in _warn_on_inexact_decimal_tolerance
if not inexact or not _contains_decimal(expected):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.14/site-packages/_pytest/approx.py:1036: in _contains_decimal
values = getattr(expected, "flat", expected)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Array(42, dtype=int32, weak_type=True)
def _notimplemented_flat(self):
"""Not implemented: Use :meth:`~jax.Array.flatten` instead."""
> raise NotImplementedError("JAX Arrays do not implement the arr.flat property: "
"consider arr.flatten() instead.")
E NotImplementedError: JAX Arrays do not implement the arr.flat property: consider arr.flatten() instead.
.venv/lib/python3.14/site-packages/jax/_src/numpy/array_methods.py:650: NotImplementedError
=================================================== short test summary info ===================================================
FAILED test.py::test_jax_approx - NotImplementedError: JAX Arrays do not implement the arr.flat property: consider arr.flatten() instead.
====================================================== 1 failed in 0.28s ======================================================
The cause is here:
Line 1036 tries to access the flat attribute/property of the instance of jax.Array which on this class immediately raises a NotImplementedError (and has always done so AFAICT).
uv pip list
Package Version
---------- --------
iniconfig 2.3.1
jax 0.12.0.1
jaxlib 0.12.0.1
ml-dtypes 0.6.0
numpy 2.5.4
opt-einsum 3.4.0
packaging 26.3
pluggy 1.7.0
pygments 2.21.0
pytest 9.2.0
scipy 1.18.1
pytest 9.2.0 on Ubuntu 26.04.1
The two patches
changelog/15284.bugfix.rstnew file mode 100644index 000000000..b4668d7bd@@ -0,0 +1 @@+Fixed a regression in pytest 9.2.0 where :func:`pytest.approx` raised ``NotImplementedError`` when given a float ``rel`` or ``abs`` tolerance and an array-like whose ``.flat`` is not implemented, such as a JAX array.src/_pytest/approx.pyindex 70a881e33..751e04f0b 100644@@ -1030,11 +1030,14 @@ def _contains_decimal(expected: object) -> bool: return True if isinstance(expected, Mapping): return any(isinstance(value, Decimal) for value in expected.values())+ if (np_array := _as_numpy_array(expected)) is not None:+ # Mirror approx(): array-likes are compared via their ndarray form.+ # Their own ``.flat`` cannot be relied upon (JAX arrays raise+ # NotImplementedError); the ndarray's ``.flat`` also makes a 0-d+ # array iterable, which it is not otherwise.+ return any(isinstance(value, Decimal) for value in np_array.flat) if _is_sequence_like(expected):- # ``.flat`` also makes a 0-d numpy array iterable, which it is not- # otherwise, despite being sized and subscriptable.- values = getattr(expected, "flat", expected)- return any(isinstance(value, Decimal) for value in values)+ return any(isinstance(value, Decimal) for value in expected) return False testing/python/approx.pyindex e0bfc37f8..0cd1f4bda 100644@@ -928,6 +928,44 @@ class TestApprox: assert approx(DeviceScalar(expected)) == actual assert approx(DeviceScalar(expected)) == DeviceScalar(actual) + @pytest.mark.parametrize("value", [42, [1.0, 2.0, 3.0]], ids=["0d", "1d"])+ def test_array_like_without_flat_with_rel(self, value) -> None:+ """Array-likes whose ``.flat`` raises, like JAX arrays (#15284)."""+ np = pytest.importorskip("numpy")++ class JaxLikeArray:+ def __init__(self, vals):+ self.vals = np.array(vals)++ def __array__(self, dtype=None, copy=None):+ return self.vals++ def __len__(self):+ return len(self.vals)++ def __getitem__(self, item):+ return self.vals[item]++ @property+ def flat(self):+ raise NotImplementedError("arrays do not implement arr.flat")++ assert JaxLikeArray(value) == approx(JaxLikeArray(value), rel=1e-3)+ assert JaxLikeArray(value) == approx(JaxLikeArray(value), abs=0.01)++ def test_array_like_of_decimals_warns_on_inexact_tolerance(self) -> None:+ np = pytest.importorskip("numpy")++ class DecimalArray:+ def __array__(self, dtype=None, copy=None):+ return np.array([Decimal("2.60")], dtype=object)++ with pytest.warns(+ pytest.PytestApproxDecimalToleranceWarning,+ match=r"rel=0\.01 cannot be represented exactly",+ ):+ approx(DecimalArray(), rel=0.01)+ def test_doctests(self, mocked_doctest_runner) -> None: import doctest
AUTHORSindex 9e4327c77..136ffd6c5 100644@@ -187,6 +187,7 @@ Floris Bruynooghe Frank Hoffmann Fraser Stark Freya Bruhin+Gabriel Gerlero Gabriel Landau Gabriel Reis Gang Eun Leechangelog/15284.bugfix.rstnew file mode 100644index 000000000..51e828405@@ -0,0 +1 @@+Fixed a regression in :func:`pytest.approx` with JAX arrays when an inexact float tolerance is supplied.src/_pytest/approx.pyindex 70a881e33..10c88d423 100644@@ -1033,7 +1033,10 @@ def _contains_decimal(expected: object) -> bool: if _is_sequence_like(expected): # ``.flat`` also makes a 0-d numpy array iterable, which it is not # otherwise, despite being sized and subscriptable.- values = getattr(expected, "flat", expected)+ try:+ values = getattr(expected, "flat", expected)+ except NotImplementedError: # ``.flat`` raises like this on a jax.Array+ return False # But JAX arrays cannot contain `Decimal` anyway return any(isinstance(value, Decimal) for value in values) return False testing/python/approx.pyindex e0bfc37f8..35adcfea1 100644@@ -898,6 +898,41 @@ class TestApprox: # see issue #12114 for test case assert vec1 != approx(vec2) + @pytest.mark.parametrize("values", (42, [42, 43], [[42, 43], [44, 45]]))+ def test_custom_array_unimplemented_flat(self, values) -> None:+ """Test handling of an array which raises NotImplementedError from+ `flat`, as is done by JAX (#15284)."""+ np = pytest.importorskip("numpy")++ class CustomArray:+ def __init__(self, values, /):+ self.values = values+ self.array_conversions = 0++ def __array__(self, dtype=None, copy=None):+ self.array_conversions += 1+ return np.array(self.values, dtype=dtype, copy=copy)++ def __getitem__(self, _):+ raise AssertionError("Should not be called") # pragma: no cover++ def __len__(self):+ raise AssertionError("Should not be called") # pragma: no cover++ @property+ def flat(self):+ raise NotImplementedError("this class does not provide .flat")++ actual = np.array(values)+ expected = CustomArray(values)++ assert actual == approx(expected, rel=1e-3)+ assert expected.array_conversions == 1+ assert actual + 1e-4 == approx(expected, rel=1e-3)+ assert expected.array_conversions == 2+ assert actual + 1 != approx(expected, rel=1e-3)+ assert expected.array_conversions == 3+ def test_numpy_array_protocol(self): """ array-like objects such as tensorflow's DeviceArray are handled like ndarray.