Backtestspytestrun-20261106-234554#15284

Issue #15284 Merged fix PR #15285 Run 6 Nov 2026 · run-20261106-234554 pytest-dev/pytest

pytest.approx: 9.2.0 broke compatibility with JAX arrays

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches fix the reported NotImplementedError, and the independent tests pass identically for both. The difference is in root cause vs. symptom. Patch A wraps the `.flat` lookup in `try/except NotImplementedError`, which is keyed to JAX's specific failure mode: any other array-like whose `.flat` raises a different exception (or returns something non-iterable) would break again, and the fix leaves `_contains_decimal` inspecting a foreign object's `.flat` even though `approx()` itself never compares the object that way. Patch B removes that reliance entirely by routing array-likes through `_as_numpy_array`, the same dispatch `approx()` uses for comparison, so the Decimal check now inspects exactly the ndarray that will be compared. That also makes the warning actually fire for array-likes of Decimals, which B covers with a dedicated test, and it drops the confusing `getattr(expected, 'flat', expected)` trick in favour of plain iteration for ordinary sequences. B's regression test covers 0-d and 1-d plus both `rel` and `abs`; A's covers 0-d/1-d/2-d and in/out of tolerance, so test quality is comparable. The main cost of B is one extra `__array__` conversion at approx construction time (a device-to-host copy for JAX), which is a performance tradeoff rather than a correctness problem; A's test explicitly asserts a single conversion, showing it avoids that. On balance B's structural fix is more robust and consistent with the surrounding code, while A is a narrower special case, so B wins on correctness and craft despite the extra conversion.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

3m 21s

Wall-clock time in the container

Full test suite

No new failures

761 passed · 4 skipped · 765 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

50vs 42 merged

Changed lines, ours vs merged

<!--
Thanks for submitting an issue!

Quick check-list while reporting bugs:
-->

  • a detailed description of the bug or problem you are having
  • output of pip list from the virtual environment you are using
  • pytest and operating system versions
  • minimal example if possible

Description

#15006 broke compatibility with JAX arrays when used with pytest.approx.

Minimal example

import jax.numpy as jnp
import pytest

def test_jax_approx():
    assert jnp.array(42) == pytest.approx(jnp.array(42), rel=1e-3)

With pytest 9.1.1: OK

With pytest 9.2.0:

========================================================== FAILURES ===========================================================
_______________________________________________________ test_jax_approx _______________________________________________________

    def test_jax_approx():
>       assert jnp.array(42) == pytest.approx(jnp.array(42), rel=1e-3)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.14/site-packages/_pytest/approx.py:1062: in _warn_on_inexact_decimal_tolerance
    if not inexact or not _contains_decimal(expected):
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.14/site-packages/_pytest/approx.py:1036: in _contains_decimal
    values = getattr(expected, "flat", expected)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Array(42, dtype=int32, weak_type=True)

    def _notimplemented_flat(self):
      """Not implemented: Use :meth:`~jax.Array.flatten` instead."""
>     raise NotImplementedError("JAX Arrays do not implement the arr.flat property: "
                                "consider arr.flatten() instead.")
E     NotImplementedError: JAX Arrays do not implement the arr.flat property: consider arr.flatten() instead.

.venv/lib/python3.14/site-packages/jax/_src/numpy/array_methods.py:650: NotImplementedError
=================================================== short test summary info ===================================================
FAILED test.py::test_jax_approx - NotImplementedError: JAX Arrays do not implement the arr.flat property: consider arr.flatten() instead.
====================================================== 1 failed in 0.28s ======================================================

The cause is here:

https://github.com/pytest-dev/pytest/blob/a7b2446ee41a7298d3a0bc22c52f3b8bacae1566/src/_pytest/approx.py#L1033-L1037

Line 1036 tries to access the flat attribute/property of the instance of jax.Array which on this class immediately raises a NotImplementedError (and has always done so AFAICT).

uv pip list

Package    Version
---------- --------
iniconfig  2.3.1
jax        0.12.0.1
jaxlib     0.12.0.1
ml-dtypes  0.6.0
numpy      2.5.4
opt-einsum 3.4.0
packaging  26.3
pluggy     1.7.0
pygments   2.21.0
pytest     9.2.0
scipy      1.18.1

pytest 9.2.0 on Ubuntu 26.04.1

The two patches

Our patch+46 −4
changelog/15284.bugfix.rstnew file mode 100644index 000000000..b4668d7bd@@ -0,0 +1 @@+Fixed a regression in pytest 9.2.0 where :func:`pytest.approx` raised ``NotImplementedError`` when given a float ``rel`` or ``abs`` tolerance and an array-like whose ``.flat`` is not implemented, such as a JAX array.src/_pytest/approx.pyindex 70a881e33..751e04f0b 100644@@ -1030,11 +1030,14 @@ def _contains_decimal(expected: object) -> bool:         return True     if isinstance(expected, Mapping):         return any(isinstance(value, Decimal) for value in expected.values())+    if (np_array := _as_numpy_array(expected)) is not None:+        # Mirror approx(): array-likes are compared via their ndarray form.+        # Their own ``.flat`` cannot be relied upon (JAX arrays raise+        # NotImplementedError); the ndarray's ``.flat`` also makes a 0-d+        # array iterable, which it is not otherwise.+        return any(isinstance(value, Decimal) for value in np_array.flat)     if _is_sequence_like(expected):-        # ``.flat`` also makes a 0-d numpy array iterable, which it is not-        # otherwise, despite being sized and subscriptable.-        values = getattr(expected, "flat", expected)-        return any(isinstance(value, Decimal) for value in values)+        return any(isinstance(value, Decimal) for value in expected)     return False  testing/python/approx.pyindex e0bfc37f8..0cd1f4bda 100644@@ -928,6 +928,44 @@ class TestApprox:         assert approx(DeviceScalar(expected)) == actual         assert approx(DeviceScalar(expected)) == DeviceScalar(actual) +    @pytest.mark.parametrize("value", [42, [1.0, 2.0, 3.0]], ids=["0d", "1d"])+    def test_array_like_without_flat_with_rel(self, value) -> None:+        """Array-likes whose ``.flat`` raises, like JAX arrays (#15284)."""+        np = pytest.importorskip("numpy")++        class JaxLikeArray:+            def __init__(self, vals):+                self.vals = np.array(vals)++            def __array__(self, dtype=None, copy=None):+                return self.vals++            def __len__(self):+                return len(self.vals)++            def __getitem__(self, item):+                return self.vals[item]++            @property+            def flat(self):+                raise NotImplementedError("arrays do not implement arr.flat")++        assert JaxLikeArray(value) == approx(JaxLikeArray(value), rel=1e-3)+        assert JaxLikeArray(value) == approx(JaxLikeArray(value), abs=0.01)++    def test_array_like_of_decimals_warns_on_inexact_tolerance(self) -> None:+        np = pytest.importorskip("numpy")++        class DecimalArray:+            def __array__(self, dtype=None, copy=None):+                return np.array([Decimal("2.60")], dtype=object)++        with pytest.warns(+            pytest.PytestApproxDecimalToleranceWarning,+            match=r"rel=0\.01 cannot be represented exactly",+        ):+            approx(DecimalArray(), rel=0.01)+     def test_doctests(self, mocked_doctest_runner) -> None:         import doctest  
Merged fix · PR #15285+41 −1
AUTHORSindex 9e4327c77..136ffd6c5 100644@@ -187,6 +187,7 @@ Floris Bruynooghe Frank Hoffmann Fraser Stark Freya Bruhin+Gabriel Gerlero Gabriel Landau Gabriel Reis Gang Eun Leechangelog/15284.bugfix.rstnew file mode 100644index 000000000..51e828405@@ -0,0 +1 @@+Fixed a regression in :func:`pytest.approx` with JAX arrays when an inexact float tolerance is supplied.src/_pytest/approx.pyindex 70a881e33..10c88d423 100644@@ -1033,7 +1033,10 @@ def _contains_decimal(expected: object) -> bool:     if _is_sequence_like(expected):         # ``.flat`` also makes a 0-d numpy array iterable, which it is not         # otherwise, despite being sized and subscriptable.-        values = getattr(expected, "flat", expected)+        try:+            values = getattr(expected, "flat", expected)+        except NotImplementedError:  # ``.flat`` raises like this on a jax.Array+            return False  # But JAX arrays cannot contain `Decimal` anyway         return any(isinstance(value, Decimal) for value in values)     return False testing/python/approx.pyindex e0bfc37f8..35adcfea1 100644@@ -898,6 +898,41 @@ class TestApprox:         # see issue #12114 for test case         assert vec1 != approx(vec2) +    @pytest.mark.parametrize("values", (42, [42, 43], [[42, 43], [44, 45]]))+    def test_custom_array_unimplemented_flat(self, values) -> None:+        """Test handling of an array which raises NotImplementedError from+        `flat`, as is done by JAX (#15284)."""+        np = pytest.importorskip("numpy")++        class CustomArray:+            def __init__(self, values, /):+                self.values = values+                self.array_conversions = 0++            def __array__(self, dtype=None, copy=None):+                self.array_conversions += 1+                return np.array(self.values, dtype=dtype, copy=copy)++            def __getitem__(self, _):+                raise AssertionError("Should not be called")  # pragma: no cover++            def __len__(self):+                raise AssertionError("Should not be called")  # pragma: no cover++            @property+            def flat(self):+                raise NotImplementedError("this class does not provide .flat")++        actual = np.array(values)+        expected = CustomArray(values)++        assert actual == approx(expected, rel=1e-3)+        assert expected.array_conversions == 1+        assert actual + 1e-4 == approx(expected, rel=1e-3)+        assert expected.array_conversions == 2+        assert actual + 1 != approx(expected, rel=1e-3)+        assert expected.array_conversions == 3+     def test_numpy_array_protocol(self):         """         array-like objects such as tensorflow's DeviceArray are handled like ndarray.