Backtestspytestrun-20261106-234554#9924
Leakage of NUMBER option out of docstring scope
The merged fix was judged better than our patch.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the same root cause the same way: pytest's runner raises out of DocTestRunner.run (on failure without --doctest-continue-on-failure, on exceptions, and on skip/xfail outcomes) before CPython's __run reaches its `self.optionflags = original_optionflags` restore, so per-example directives leak into later doctests sharing the module runner. Patch A saves/restores around `self.runner.run` in `DoctestItem.runtest`; Patch B overrides `run` in the custom runner class. Since runtest is the only caller of run, the two are functionally equivalent, and the independent tests confirm both pass every scenario (including exception and continue-on-failure). B's placement is slightly more self-contained, but A's placement is equally correct and simpler. The difference is in tests. Patch A adds two parametrized regression tests that use `assert_outcomes` with exact counts, cover failure and exception in the first doctest, cover enabling NUMBER, enabling ELLIPSIS, and disabling ELLIPSIS (the opposite direction of leakage, where a leaked `-ELLIPSIS` would make a later test wrongly fail), and separately cover skip and xfail outcomes, which also bypass doctest's restore. Each of these would fail before the fix and pass after. Patch B's single test covers failure/exception/skip with +NUMBER only and asserts merely that some failing nodeid ends with the second test's name, a weaker check that does not pin down total outcomes. Neither test is skipped at runtime (the `pytest.skip` in B is inside a doctest example, not the test itself). Scope and craft are comparable: both add a one-line changelog and an accurate comment; B's changelog is more specific, A's mentions skip/xfail. On balance A wins narrowly on more thorough, precise test coverage with an equivalent fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
3m 33s
Wall-clock time in the container
Full test suite
No new failures
149 passed · 2 skipped · 151 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
56vs 94 merged
Changed lines, ours vs merged
The issue
View on GitHub<!--
Thanks for submitting an issue!
Quick check-list while reporting bugs:
-->
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible
This was originally noted in #9892, but there is a more targeted MRE here.
MRE
Running pytest --doctest-modules test_x.py with x=1, 2 or 3 results in the following pattern of failing and passing tests. The test inside test_wo_number incorrectly passes only on the cases (test_1.py and test_2.py) where a failing test that uses NUMBER (test_w_off_by_2_number) also exists in the same module. This leakage does not occur when there are tests that pass that utilize NUMBER (test_3.py)
# test_1.py
def test_w_number():
"""
Correctly Passes
Examples
--------
>>> 1. # doctest: +NUMBER
0.
"""
def test_w_off_by_2_number():
"""
Correctly Fails
Examples
--------
>>> 0. # doctest: +NUMBER
2.
"""
def test_wo_number():
"""
Incorrectly Passes
Examples
--------
>>> 1.
0.
"""
# test_2.py
def test_w_off_by_2_number():
"""
Correctly Fails
Examples
--------
>>> 0. # doctest: +NUMBER
2.
"""
def test_wo_number():
"""
Incorrectly Passes
Examples
--------
>>> 1.
0.
"""
# test_3.py
def test_w_number():
"""
Correctly Passes
Examples
--------
>>> 1. # doctest: +NUMBER
0.
"""
def test_wo_number():
"""
Correctly Fails
Examples
--------
>>> 1.
0.
"""
pip list
Package Version
---------- -------
attrs 21.4.0
iniconfig 1.1.1
numpy 1.22.3
packaging 21.3
pip 22.0.4
pluggy 1.0.0
py 1.11.0
pyparsing 3.0.8
pytest 7.1.2
setuptools 41.2.0
tomli 2.0.1
system version
CentOS 7.9
The two patches
changelog/9924.bugfix.rstnew file mode 100644index 000000000..2e2872f09@@ -0,0 +1 @@+Fixed doctest option directives such as ``+NUMBER`` leaking into the following doctests of the same module when a doctest stopped early, e.g. on failure without ``--doctest-continue-on-failure``.src/_pytest/doctest.pyindex ea894b3f4..5992c9679 100644@@ -196,6 +196,24 @@ def _init_runner_class() -> type[doctest.DocTestRunner]: super().__init__(checker=checker, verbose=verbose, optionflags=optionflags) self.continue_on_failure = continue_on_failure + def run(+ self,+ test: doctest.DocTest,+ compileflags: int | None = None,+ out=None,+ clear_globs: bool = True,+ ) -> doctest.TestResults:+ # The runner is shared between all doctests of a module/file, and+ # DocTestRunner only restores the optionflags modified by per-example+ # directives (e.g. ``# doctest: +NUMBER``) if the run completes+ # normally; when a failure is raised mid-run (#9924) they would leak+ # into the following doctests, so restore them here.+ original_optionflags = self.optionflags+ try:+ return super().run(test, compileflags, out, clear_globs)+ finally:+ self.optionflags = original_optionflags+ def report_failure( self, out,testing/test_doctest.pyindex a2b91bc40..33393714b 100644@@ -1199,6 +1199,43 @@ class TestLiterals: reprec = pytester.inline_run() reprec.assertoutcome(passed=1) + @pytest.mark.parametrize(+ "first_examples",+ [+ pytest.param([">>> 0. # doctest: +NUMBER", "2."], id="failure"),+ pytest.param([">>> 1 / 0 # doctest: +NUMBER"], id="exception"),+ pytest.param(+ [">>> import pytest; pytest.skip('skipped') # doctest: +NUMBER"],+ id="skip",+ ),+ ],+ )+ @pytest.mark.parametrize("continue_on_failure", [False, True])+ def test_number_does_not_leak_to_other_doctests(+ self, pytester: Pytester, first_examples: list[str], continue_on_failure: bool+ ) -> None:+ """Option directives of an example which stops the run must not leak+ into the following doctests of the same module (#9924)."""+ first_body = "".join(f" {line}\n" for line in first_examples)+ pytester.makepyfile(+ "def test_a_with_number():\n"+ ' """\n'+ f"{first_body}"+ ' """\n'+ "\n"+ "def test_b_without_number():\n"+ ' """\n'+ " >>> 1.\n"+ " 0.\n"+ ' """\n'+ )+ args = ["--doctest-modules"]+ if continue_on_failure:+ args.append("--doctest-continue-on-failure")+ reprec = pytester.inline_run(*args)+ failed = [rep.nodeid for rep in reprec.getfailures()]+ assert any(nodeid.endswith(".test_b_without_number") for nodeid in failed)+ class TestDoctestSkips: """
changelog/9924.bugfix.rstnew file mode 100644index 000000000..9e26bc6c5@@ -0,0 +1 @@+Fixed doctest option flags leaking between docstrings after a failure, skip, or expected failure.src/_pytest/doctest.pyindex ea894b3f4..04d707113 100644@@ -297,9 +297,14 @@ class DoctestItem(Item): _check_all_skipped(self.dtest) self._disable_output_capturing_for_darwin() failures: list[doctest.DocTestFailure] = []- # Type ignored because we change the type of `out` from what- # doctest expects.- self.runner.run(self.dtest, out=failures) # type: ignore[arg-type]+ optionflags = self.runner.optionflags+ try:+ # Type ignored because we change the type of `out` from what+ # doctest expects.+ self.runner.run(self.dtest, out=failures) # type: ignore[arg-type]+ finally:+ # Our runner can raise before doctest restores its option flags.+ self.runner.optionflags = optionflags if failures: raise MultipleDoctestFailures(failures) testing/test_doctest.pyindex a2b91bc40..a322e9eb2 100644@@ -272,6 +272,35 @@ class TestDoctests: ] ) + @pytest.mark.parametrize("outcome", ["skip", "xfail"])+ @pytest.mark.parametrize("continue_on_failure", [False, True])+ def test_optionflags_restored_after_outcome(+ self, pytester: Pytester, outcome: str, continue_on_failure: bool+ ) -> None:+ pytester.makepyfile(+ f"""+ import pytest++ def first():+ '''+ >>> pytest.{outcome}("reason") # doctest: -ELLIPSIS+ '''++ def second():+ '''+ >>> print("foobar")+ foo...+ '''+ """+ )+ args = ["--doctest-modules"]+ if continue_on_failure:+ args.append("--doctest-continue-on-failure")+ result = pytester.runpytest(*args)+ result.assert_outcomes(+ passed=1, skipped=int(outcome == "skip"), xfailed=int(outcome == "xfail")+ )+ def test_docstring_partial_context_around_error(self, pytester: Pytester): """Test that we show some context before the actual line of a failing doctest.@@ -679,6 +708,59 @@ class TestDoctests: reprec = pytester.inline_run(p, "--doctest-modules") reprec.assertoutcome(failed=1, passed=1) + @pytest.mark.parametrize("continue_on_failure", [False, True])+ @pytest.mark.parametrize("first_example", ["0.", "1 / 0"])+ @pytest.mark.parametrize(+ "optionflags, directive, second_example, second_output, passed",+ [+ pytest.param("ELLIPSIS", "+NUMBER", "1.", "0.", 0, id="enable-number"),+ pytest.param(+ "", "+ELLIPSIS", 'print("foobar")', "foo...", 0, id="enable-ellipsis"+ ),+ pytest.param(+ "ELLIPSIS",+ "-ELLIPSIS",+ 'print("foobar")',+ "foo...",+ 1,+ id="disable-ellipsis",+ ),+ ],+ )+ def test_optionflags_do_not_leak_between_docstrings(+ self,+ pytester: Pytester,+ continue_on_failure: bool,+ first_example: str,+ optionflags: str,+ directive: str,+ second_example: str,+ second_output: str,+ passed: int,+ ) -> None:+ """A failing docstring must not change another's comparison options (#9924)."""+ pytester.makeini(f"[pytest]\ndoctest_optionflags = {optionflags}")+ pytester.makepyfile(+ f"""+ def first():+ '''+ >>> {first_example} # doctest: {directive}+ 2.+ '''++ def second():+ '''+ >>> {second_example}+ {second_output}+ '''+ """+ )+ args = ["--doctest-modules"]+ if continue_on_failure:+ args.append("--doctest-continue-on-failure")+ result = pytester.runpytest(*args)+ result.assert_outcomes(failed=2 - passed, passed=passed)+ def test_ignored_whitespace(self, pytester: Pytester): pytester.makeini( """