Backtestspytestrun-20261106-234554#14412
Incorrect timing info when using subtests and console_output_style='times'
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The issue's reproducer is a unittest.TestCase using self.subTest, and the reported symptom is that subtest lines show 0.000us. Patch B only fixes the terminal reporter: it returns the SubtestReport's duration in verbose mode and excludes subtest reports from the per-module accounting. But for unittest subtests, _pytest/unittest.py builds the CallInfo with duration=0, so B still prints 0.000us for every unittest SUBPASSED line. The independent test test_unittest_subtest_lines_show_subtest_duration confirms this: it fails on B with 'SUBPASSED(i=0) 0.000us', while A passes. Patch A fixes both halves: it threads the current report into _get_progress_information_message (cleaner than B's instance-level _current_logreport side channel), shows the subtest's own duration on subtest lines, excludes subtests from the module sum so their time is not double-counted, and wraps TestCase.subTest to measure start/stop so addSubTest produces a real duration, with cleanup of the wrapper and timing dict after the test. A's tests cover the unittest case (passing and failing), the subtests fixture case, and the once-counted module time including line wrap, with actual lower-bound duration assertions. B's tests only use the subtests fixture, assert merely that '0.000us' does not appear, include an xdist test that skips when xdist is absent, and add an unrelated slow-teardown test and AUTHORS churn. A resolves the issue at its root cause; B masks only the fixture path and leaves the issue's exact scenario broken.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
13m 14s
Wall-clock time in the container
Full test suite
No new failures
366 passed · 1 skipped · 367 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
284vs 166 merged
Changed lines, ours vs merged
The issue
View on GitHubpytest does not seem to report accurate timing info when using subtests and console_output_style = 'times':
pyproject.toml
[tool.pytest]
addopts = [
'--capture=tee-sys',
'-v'
]
console_output_style = 'times'
test_subtests.py
import time
import unittest
class Test(unittest.TestCase):
def test_subtests(self):
for i in range(3):
with self.subTest(i=i):
time.sleep(1)
Output:
$ pytest
==================================== test session starts =====================================
platform linux -- Python 3.14.4, pytest-9.1.0.dev332+gd4fde45a5, pluggy-1.6.0 -- /home/user/.venv-pyt/bin/python
cachedir: .pytest_cache
rootdir: /home/user/scratch
configfile: pyproject.toml
collected 1 item
test_subtests.py::Test::test_subtests SUBPASSED(i=0) 570.0us
test_subtests.py::Test::test_subtests SUBPASSED(i=1) 0.000us
test_subtests.py::Test::test_subtests SUBPASSED(i=2) 0.000us
test_subtests.py::Test::test_subtests PASSED 0.000us
============================ 1 passed, 3 subtests passed in 3.02s ============================
Version info
Package Version Editable project location
--------- ----------------------- -------------------------
iniconfig 2.3.0
packaging 26.1
pip 26.0.1
pluggy 1.6.0
Pygments 2.20.0
pytest 9.1.0.dev332+gd4fde45a5 /home/user/code/pytest
- pytest 9.0.3 and d4fde45a5
- Python 3.14.4
- Arch Linux
The two patches
changelog/14412.bugfix.rstnew file mode 100644index 000000000..633ba8739@@ -0,0 +1,2 @@+Fixed the durations shown with :confval:`console_output_style` set to ``times`` when using subtests: each subtest line now shows the duration of that subtest, the test line shows the duration of the whole test, and the duration of subtests is no longer counted twice (or dropped) in the per-module time.+Subtests of :class:`unittest.TestCase` tests now also report their duration, instead of always reporting zero.src/_pytest/terminal.pyindex 023fdcaab..6282ff1bc 100644@@ -409,7 +409,9 @@ class TerminalReporter: # We use CallableBool here to support both. self.isatty = compat.CallableBool(file.isatty()) self._progress_nodeids_reported: set[NodeId] = set()- self._timing_nodeids_reported: set[NodeId] = set()+ # Identities of the reports whose durations were already displayed+ # with console_output_style=times.+ self._timing_reports_reported: set[int] = set() self._show_progress_info = self._determine_show_progress_info() self._collect_report_last_write = timing.Instant() self._already_displayed_warnings: int | None = None@@ -694,13 +696,13 @@ class TerminalReporter: if reason and formatted_reason is not None: self.wrap_write(formatted_reason) if self._show_progress_info:- self._write_progress_information_filling_space()+ self._write_progress_information_filling_space(rep) else: self.ensure_newline() self._tw.write(f"[{rep.node.gateway.id}]") if self._show_progress_info: self._tw.write(- self._get_progress_information_message() + " ", cyan=True+ self._get_progress_information_message(rep) + " ", cyan=True ) else: self._tw.write(" ")@@ -728,7 +730,15 @@ class TerminalReporter: return result - def _get_progress_information_message(self) -> str:+ def _get_progress_information_message(+ self, report: TestReport | None = None+ ) -> str:+ """Return the progress information to display.++ :param report:+ The report whose result line is being written, when each report+ gets its own line (verbose mode).+ """ assert self._session collected = self._session.testscollected if self._show_progress_info == "count":@@ -741,6 +751,12 @@ class TerminalReporter: if self._show_progress_info == "times": if not collected: return ""+ # Imported here to avoid a circular import.+ from _pytest.subtests import SubtestReport++ if isinstance(report, SubtestReport):+ # A subtest line shows the duration of that subtest only.+ return format_node_duration(report.duration) all_reports = ( self._get_reports_to_display("passed") + self._get_reports_to_display("xpassed")@@ -750,25 +766,36 @@ class TerminalReporter: + self._get_reports_to_display("error") + self._get_reports_to_display("") )- current_location = all_reports[-1].location[0]+ # Subtest reports share the nodeid of their test, and their+ # durations are already part of the duration of the test's call+ # phase, so they must not be accounted for separately. not_reported = [- r for r in all_reports if r.id not in self._timing_nodeids_reported+ r+ for r in all_reports+ if id(r) not in self._timing_reports_reported+ and not isinstance(r, SubtestReport) ]- tests_in_module = sum(- i.location[0] == current_location for i in self._session.items- )- tests_completed = sum(- r.when == "setup"- for r in not_reported- if r.location[0] == current_location- )- last_in_module = tests_completed == tests_in_module- if self.showlongtestinfo or last_in_module:- self._timing_nodeids_reported.update(r.id for r in not_reported)- return format_node_duration(- sum(r.duration for r in not_reported if isinstance(r, TestReport))+ if report is not None:+ # Verbose mode: one line per test.+ not_reported = [r for r in not_reported if r.id == report.id]+ else:+ current_location = all_reports[-1].location[0]+ tests_in_module = sum(+ i.location[0] == current_location for i in self._session.items+ )+ # Count all setups, as the time of a part of the module may+ # already have been shown when the line wrapped.+ tests_completed = sum(+ r.when == "setup"+ for r in all_reports+ if r.location[0] == current_location )- return ""+ if tests_completed < tests_in_module:+ return ""+ self._timing_reports_reported.update(id(r) for r in not_reported)+ return format_node_duration(+ sum(r.duration for r in not_reported if isinstance(r, TestReport))+ ) if collected: return f" [{self.reported_progress * 100 // collected:3d}%]" return " [100%]"@@ -789,9 +816,11 @@ class TerminalReporter: msg = self._get_progress_information_message() self._tw.write(msg + "\n", **{main_color: True}) - def _write_progress_information_filling_space(self) -> None:+ def _write_progress_information_filling_space(+ self, report: TestReport | None = None+ ) -> None: color, _ = self._get_main_color()- msg = self._get_progress_information_message()+ msg = self._get_progress_information_message(report) w = self._width_of_current_line fill = self._tw.fullwidth - w - 1 self.write(msg.rjust(fill), flush=True, **{color: True})src/_pytest/unittest.pyindex 2c9e0dcb8..97b85dd24 100644@@ -7,6 +7,8 @@ from collections.abc import Callable from collections.abc import Generator from collections.abc import Iterable from collections.abc import Iterator+from contextlib import AbstractContextManager+from contextlib import contextmanager from enum import auto from enum import Enum import inspect@@ -18,6 +20,7 @@ from typing import TYPE_CHECKING from unittest import TestCase from _pytest import fixtures+from _pytest import timing import _pytest._code from _pytest._code import ExceptionInfo from _pytest.compat import assert_never@@ -248,6 +251,11 @@ class TestCaseFunction(Function): # A cache of the subTest errors and non-subtest skips in self._outcome. # Compute and cache these lists once, instead of computing them again and again for each subtest (#13965). self._cached_errors_and_skips: tuple[list[Any], list[Any]] | None = None+ # Start and stop instants of the finished subtests, keyed by the id() of+ # their unittest subtest object (also stored to keep the id valid).+ self._subtest_timings: dict[+ int, tuple[TestCase, timing.Instant, timing.Instant]+ ] = {} def teardown(self) -> None: if self._explicit_tearDown is not None:@@ -376,12 +384,54 @@ class TestCaseFunction(Function): def addDuration(self, testcase: unittest.TestCase, elapsed: float) -> None: pass - def runtest(self) -> None:- from _pytest.debugging import maybe_wrap_pytest_function_for_tracing+ def _timed_subtest(+ self,+ testcase: TestCase,+ subtest: Callable[..., AbstractContextManager[None]],+ ) -> Callable[..., AbstractContextManager[None]]:+ """Wrap ``testcase.subTest`` to measure the duration of each subtest.++ unittest only notifies the result object when a subtest finishes (and+ on Python < 3.11, only once the whole test has finished), so the time+ is measured here and used by `addSubTest()`.+ """++ @contextmanager+ def timed_subtest(*args: Any, **kwargs: Any) -> Iterator[None]:+ with subtest(*args, **kwargs):+ current = testcase._subtest # type: ignore[attr-defined]+ start = timing.Instant()+ try:+ yield+ finally:+ self._subtest_timings[id(current)] = (+ current,+ start,+ timing.Instant(),+ )++ return timed_subtest + def runtest(self) -> None: testcase = self.instance assert testcase is not None + original_subtest = testcase.__dict__.get("subTest")+ setattr(testcase, "subTest", self._timed_subtest(testcase, testcase.subTest))+ try:+ self._runtest(testcase)+ finally:+ # Entries of subtests never reported (e.g. under expectedFailure)+ # would otherwise keep the TestCase instance alive (#1649).+ self._subtest_timings.clear()+ if original_subtest is None:+ delattr(testcase, "subTest")+ else:+ setattr(testcase, "subTest", original_subtest)++ def _runtest(self, testcase: TestCase) -> None:+ from _pytest.debugging import maybe_wrap_pytest_function_for_tracing+ maybe_wrap_pytest_function_for_tracing(self) # Let the unittest framework handle async functions.@@ -441,12 +491,18 @@ class TestCaseFunction(Function): case unreachable: assert_never(unreachable) + timings = self._subtest_timings.pop(id(test), None)+ if timings is not None and timings[0] is test:+ _, start, stop = timings+ else:+ # Subtest not entered through the subTest() wrapper (see `runtest()`).+ start = stop = timing.Instant() call_info = CallInfo[None]( None, exception_info,- start=0,- stop=0,- duration=0,+ start=start.time,+ stop=stop.time,+ duration=timing.Duration(start=start, stop=stop).seconds, when="call", _ispytest=True, )testing/test_terminal.pyindex 30208084a..b2c5b7098 100644@@ -6,6 +6,7 @@ from __future__ import annotations from io import StringIO import os from pathlib import Path+import re import sys import textwrap from types import SimpleNamespace@@ -2352,6 +2353,148 @@ class TestProgressOutputStyle: ] ) + @staticmethod+ def _line_duration(line: str) -> float:+ """Parse the duration (in seconds) displayed at the end of a line by+ console_output_style=times (see ``format_node_duration``)."""+ m = re.search(r"(\d+(?:\.\d+)?)(us|ms|s)$", line)+ assert m is not None, line+ value, unit = float(m.group(1)), m.group(2)+ return value * {"us": 1e-6, "ms": 1e-3, "s": 1.0}[unit]++ @pytest.mark.parametrize("failing", [False, True], ids=["passing", "failing"])+ def test_verbose_times_unittest_subtests(+ self, pytester: Pytester, failing: bool+ ) -> None:+ """Each subtest line shows the duration of the subtest, and the test+ line shows the duration of the whole test (#14412)."""+ pytester.makeini(+ """+ [pytest]+ console_output_style = times+ """+ )+ pytester.makepyfile(+ test_timed_subtests=f"""+ import time+ import unittest++ class Test(unittest.TestCase):+ def test_subtests(self):+ for i in range(3):+ with self.subTest(i=i):+ time.sleep(0.2)+ assert not {failing}+ """+ )+ result = pytester.runpytest("-v")+ status = "SUBFAILED" if failing else "SUBPASSED"+ result.stdout.re_match_lines(+ [+ rf"test_timed_subtests.py::Test::test_subtests {status}\(i=0\) .*",+ rf"test_timed_subtests.py::Test::test_subtests {status}\(i=1\) .*",+ rf"test_timed_subtests.py::Test::test_subtests {status}\(i=2\) .*",+ r"test_timed_subtests.py::Test::test_subtests (PASSED|FAILED) .*",+ ]+ )+ lines = [+ line+ for line in result.stdout.lines+ if line.startswith("test_timed_subtests.py::Test::test_subtests ")+ ]+ assert len(lines) == 4, lines+ for line in lines[:3]:+ assert self._line_duration(line) >= 0.2, line+ assert self._line_duration(lines[3]) >= 0.6, lines[3]++ def test_verbose_times_subtests_fixture(self, pytester: Pytester) -> None:+ pytester.makeini(+ """+ [pytest]+ console_output_style = times+ """+ )+ pytester.makepyfile(+ test_timed_subtests="""+ import time++ def test_subtests(subtests):+ for i in range(2):+ with subtests.test(i=i):+ time.sleep(0.2)+ assert i == 0++ def test_after():+ pass+ """+ )+ result = pytester.runpytest("-v")+ result.stdout.re_match_lines(+ [+ r"test_timed_subtests.py::test_subtests SUBPASSED\(i=0\) .*",+ r"test_timed_subtests.py::test_subtests SUBFAILED\(i=1\) .*",+ r"test_timed_subtests.py::test_subtests FAILED .*",+ r"test_timed_subtests.py::test_after PASSED .*",+ ]+ )+ lines = {+ line.split(" ", 2)[1]: line+ for line in result.stdout.lines+ if line.startswith("test_timed_subtests.py::")+ }+ assert self._line_duration(lines["SUBPASSED(i=0)"]) >= 0.2+ assert self._line_duration(lines["SUBFAILED(i=1)"]) >= 0.2+ assert self._line_duration(lines["FAILED"]) >= 0.4+ assert self._line_duration(lines["PASSED"]) < 0.2++ @pytest.mark.parametrize("columns", [80, 40], ids=["one-line", "multiline"])+ def test_times_subtests_counted_once(+ self, pytester: Pytester, monkeypatch: MonkeyPatch, columns: int+ ) -> None:+ """The module time counts the duration of the subtests once, as they+ are part of the test call, even when the progress line wraps (#14412)."""+ monkeypatch.setenv("COLUMNS", str(columns))+ pytester.makeini(+ """+ [pytest]+ console_output_style = times+ """+ )+ pytester.makepyfile(+ # Short module name, so that the line wraps during the subtests.+ test_ts="""+ import time+ import pytest++ @pytest.mark.parametrize("n", range(10))+ def test_quick(n):+ pass++ def test_subtests(subtests):+ for i in range(20):+ with subtests.test(i=i):+ time.sleep(0.05)+ assert False+ """,+ # Not the last module, so that the line can wrap during the subtests.+ test_z="""+ def test_z():+ pass+ """,+ )+ result = pytester.runpytest()+ result.assert_outcomes(passed=11, failed=21)+ lines = [+ line+ for line in result.stdout.lines+ if re.match(r"(test_ts\.py )?[.uF]+ +\d.*s$", line)+ ]+ assert lines, result.stdout.lines+ if columns == 40:+ assert len(lines) > 1, lines+ total = sum(self._line_duration(line) for line in lines)+ assert 1.0 <= total < 2.0, lines+ def test_xdist_normal( self, many_tests_files, pytester: Pytester, monkeypatch ) -> None:
AUTHORSindex e2fad5e83..026f6aed2 100644@@ -303,6 +303,7 @@ Marc Mueller Marc Schlaich Marcel Telka Marcelo Duarte Trevisani+Marcelo Markus Marcin Augustynów Marcin Bachry Marc Bresson@@ -454,6 +455,7 @@ Seth Junot Shantanu Jain Sharad Nair Shaygan Hooshyari+Shimon Schwartz Shubham Adep Simon Blanchard Simon Gomizeljchangelog/14412.bugfix.rstnew file mode 100644index 000000000..4148ca8b4@@ -0,0 +1 @@+Fixed incorrect timing information for subtests and parent tests when ``console_output_style="times"``.src/_pytest/terminal.pyindex 023fdcaab..45b25e7c5 100644@@ -410,6 +410,7 @@ class TerminalReporter: self.isatty = compat.CallableBool(file.isatty()) self._progress_nodeids_reported: set[NodeId] = set() self._timing_nodeids_reported: set[NodeId] = set()+ self._current_logreport: TestReport | None = None self._show_progress_info = self._determine_show_progress_info() self._collect_report_last_write = timing.Instant() self._already_displayed_warnings: int | None = None@@ -634,6 +635,7 @@ class TerminalReporter: self.flush() def pytest_runtest_logreport(self, report: TestReport) -> None:+ self._current_logreport = report self._tests_ran = True rep = report @@ -741,6 +743,13 @@ class TerminalReporter: if self._show_progress_info == "times": if not collected: return ""+ from _pytest.subtests import SubtestReport++ if isinstance(self._current_logreport, SubtestReport):+ if self.showlongtestinfo:+ return format_node_duration(self._current_logreport.duration)+ return ""+ all_reports = ( self._get_reports_to_display("passed") + self._get_reports_to_display("xpassed")@@ -750,9 +759,18 @@ class TerminalReporter: + self._get_reports_to_display("error") + self._get_reports_to_display("") )- current_location = all_reports[-1].location[0]++ report = self._current_logreport+ if report is not None:+ current_location = report.location[0]+ else:+ current_location = all_reports[-1].location[0] if all_reports else ""+ not_reported = [- r for r in all_reports if r.id not in self._timing_nodeids_reported+ r+ for r in all_reports+ if not isinstance(r, SubtestReport)+ and r.id not in self._timing_nodeids_reported ] tests_in_module = sum( i.location[0] == current_location for i in self._session.itemstesting/test_terminal.pyindex 30208084a..e77e265f9 100644@@ -140,6 +140,147 @@ class TestTerminal: combined = "\n".join(result.stdout.lines + result.stderr.lines) assert "INTERNALERROR" not in combined + def test_console_output_style_times_with_subtests(self, pytester: Pytester) -> None:+ pytester.makepyfile(+ test_repro="""+ def test_subtests(subtests):+ for i in range(2):+ with subtests.test(i=i):+ pass+ """+ )+ result = pytester.runpytest(+ "test_repro.py",+ "-v",+ "-o",+ "console_output_style=times",+ "-o",+ "verbosity_subtests=1",+ )++ # Check that we got positive/non-zero timing info for subtests and the parent PASSED line.+ # We check that it does not show "0.000us".+ lines = result.stdout.lines+ subpassed_lines = [+ line_content for line_content in lines if "SUBPASSED" in line_content+ ]+ passed_lines = [+ line_content+ for line_content in lines+ if "PASSED" in line_content and "SUBPASSED" not in line_content+ ]+ assert len(subpassed_lines) == 2+ assert len(passed_lines) == 1+ for line in subpassed_lines + passed_lines:+ assert "0.000us" not in line++ def test_console_output_style_times_with_subtests_xdist(+ self, pytester: Pytester, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ pytest.importorskip("xdist")+ monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", raising=False)+ pytester.makepyfile(+ test_repro="""+ def test_subtests(subtests):+ for i in range(2):+ with subtests.test(i=i):+ pass+ """+ )+ result = pytester.runpytest(+ "test_repro.py",+ "-n2",+ "-v",+ "-o",+ "console_output_style=times",+ "-o",+ "verbosity_subtests=1",+ )++ lines = result.stdout.lines+ subpassed_lines = [+ line_content for line_content in lines if "SUBPASSED" in line_content+ ]+ passed_lines = [+ line_content+ for line_content in lines+ if "PASSED" in line_content and "SUBPASSED" not in line_content+ ]+ assert len(subpassed_lines) == 2+ assert len(passed_lines) == 1+ for line in subpassed_lines + passed_lines:+ assert "0.000us" not in line++ def test_console_output_style_times_slow_teardown_does_not_leak(+ self, pytester: Pytester+ ) -> None:+ pytester.makepyfile(+ test_repro="""+ import time, pytest++ @pytest.fixture+ def slow_td():+ yield+ time.sleep(0.3)++ def test_a1(slow_td):+ pass++ def test_a2():+ pass+ """+ )+ result = pytester.runpytest(+ "test_repro.py",+ "-v",+ "-o",+ "console_output_style=times",+ )+ lines = result.stdout.lines+ a2_lines = [line for line in lines if "test_a2 PASSED" in line]+ assert len(a2_lines) == 1+ # The teardown duration of test_a1 (300ms) should not leak into test_a2.+ duration_str = a2_lines[0].split()[-1]+ if duration_str.endswith("us"):+ dur_seconds = float(duration_str[:-2]) / 1_000_000+ elif duration_str.endswith("ms"):+ dur_seconds = float(duration_str[:-2]) / 1_000+ elif duration_str.endswith("s"):+ dur_seconds = float(duration_str[:-1])+ else:+ dur_seconds = 0.0+ assert dur_seconds < 0.1++ def test_progress_information_message_no_current_report(+ self, pytester: Pytester, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ item = pytester.getitem("def test_func(): pass")+ tr = TerminalReporter(item.config)+ monkeypatch.setattr(tr.config, "get_verbosity", lambda *args, **kwargs: 1)+ tr._show_progress_info = "times"++ class MockSession:+ testscollected = 1+ items = [item]++ tr._session = MockSession() # type: ignore[assignment]++ from _pytest.reports import TestReport++ rep = TestReport(+ nodeid=item.nodeid,+ location=item.location,+ keywords={},+ outcome="passed",+ longrepr=None,+ when="call",+ duration=0.123,+ )+ tr.stats.setdefault("passed", []).append(rep)++ msg = tr._get_progress_information_message()+ assert msg == " 123.0ms"+ def test_internalerror(self, pytester: Pytester, linecomp) -> None: modcol = pytester.getmodulecol("def test_one(): pass") rep = TerminalReporter(modcol.config, file=linecomp.stringio)