Backtestspytestrun-20261106-234554#9236
LogCaptureHandler can be garbage collected when caplog fixture is in use
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The issue's trigger is an old-style hookwrapper whose teardown raises (outcome.get_result() on the failing xfail test). On pluggy < 1.6 that aborts the remaining wrapper teardowns, so the logging plugin's pytest_runtest_call wrapper is left suspended; when the garbage collector finalises it during a later test, its catching_logs contexts exit and remove the shared LogCaptureHandler from the root logger. Patch B fixes this at the root by requiring pluggy>=1.6, where all wrapper teardowns run even if one raises. That also fixes the same leak for every other plugin wrapper (capture, etc.), not just logging. Its test reproduces the issue's exact conftest/test scenario, correctly runs in a subprocess (with a clear comment on why in-process would hide the bug), and would fail on the old dependency and pass after the bump. The change is minimal and the changelog explains the mechanism. Patch A instead works around the symptom inside catching_logs with module-level reference counts keyed by (logger, handler) plus a global ordered list of contexts that hand off orig_level to each other on out-of-order exit. It does make caplog survive a GC'd leaked context, and its tests exercise that, but it leaves the actual defect (skipped teardowns) in place for everything else, adds global mutable state and branching (including an 'attached by someone else' path) that is easy to get subtly wrong, and its main tests construct the leak artificially via private plugin API rather than the issue's scenario. The evidence note that the reference fix touches only config files is consistent with B's approach. B is the more correct, better-scoped fix with a more faithful regression test.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
6m 59s
Wall-clock time in the container
Full test suite
No new failures
389 passed · 13 skipped · 406 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
178vs 50 merged
Changed lines, ours vs merged
The issue
View on GitHub- a detailed description of the bug or problem you are having
In our tests we have had issues with the caplog fixture not correctly saving every log into caplog.messages and caplog.records. From what I can tell, the underlying issue is that in some circumstances the LogCaptureHandlers that caplog relies on can get garbage collected.
There seem to be a lot of issues in this area (#5160, #8977, #7335 etc.), but I'm not sure if any of them are the specific case that I have experienced.
- minimal example if possible
Some debugging has got me to the stage where I have a MWE that reproduces the problem:
test.py:
import gc
import logging
import sys
import pytest
logging.basicConfig()
logger = logging.getLogger()
# No idea why this test must be an xfail, but we do not hit the error if this is an empty pass
@pytest.mark.xfail
def test_other_test():
assert False
def test_a_test_with_caplog(caplog):
print(logging.root.handlers)
gc.collect()
print(logging.root.handlers)
logger.error("TEST")
assert caplog.messages[0] == "TEST", caplog.messages
conftest.py:
# It is unclear if this makes a difference (as the number of objects is so small we would be
# unlikely to hit a GC anyway)
# import gc
# gc.disable()
import pytest
# Despite this being a noop, it is required to hit the issue
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
outcome = yield
outcome.get_result()
If you run pytest test.py, then it fails with the following:
❯ pytest test.py
======================================================== test session starts ========================================================
platform darwin -- Python 3.8.3, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/james/code/pytest_example
collected 2 items
test.py xF [100%]
============================================================= FAILURES ==============================================================
______________________________________________________ test_a_test_with_caplog ______________________________________________________
caplog = <_pytest.logging.LogCaptureFixture object at 0x1024ebc10>
def test_a_test_with_caplog(caplog):
print(logging.root.handlers)
gc.collect()
print(logging.root.handlers)
logger.error("TEST")
> assert caplog.messages[0] == "TEST", caplog.messages
E IndexError: list index out of range
test.py:22: IndexError
------------------------------------------------------- Captured stdout call --------------------------------------------------------
[<_LiveLoggingNullHandler (NOTSET)>, <_FileHandler /dev/null (NOTSET)>, <LogCaptureHandler (NOTSET)>, <LogCaptureHandler (NOTSET)>]
[<_LiveLoggingNullHandler (NOTSET)>, <_FileHandler /dev/null (NOTSET)>]
====================================================== short test summary info ======================================================
FAILED test.py::test_a_test_with_caplog - IndexError: list index out of range
=================================================== 1 failed, 1 xfailed in 0.04s ====================================================
However, if you remove the gc.collect() call from the test, it passes:
❯ pytest test.py
======================================================== test session starts ========================================================
platform darwin -- Python 3.8.3, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/james/code/pytest_example
collected 2 items
test.py x. [100%]
=================================================== 1 passed, 1 xfailed in 0.04s ====================================================
- output of
pip listfrom the virtual environment you are using
❯ pip list
Package Version
--------------------------------- ---------
appdirs 1.4.4
attrs 21.2.0
backports.entry-points-selectable 1.1.0
CacheControl 0.12.6
cachy 0.3.0
certifi 2021.10.8
charset-normalizer 2.0.7
cleo 0.8.1
clikit 0.6.2
crashtest 0.3.1
distlib 0.3.3
filelock 3.0.12
html5lib 1.1
idna 3.2
iniconfig 1.1.1
keyring 21.8.0
lockfile 0.12.2
msgpack 1.0.2
packaging 20.9
pastel 0.2.1
pbr 5.4.5
pexpect 4.8.0
pip 19.2.3
pkginfo 1.7.1
platformdirs 2.4.0
pluggy 1.0.0
poetry 1.1.11
poetry-core 1.0.7
ptyprocess 0.7.0
py 1.10.0
pylev 1.4.0
pyparsing 2.4.7
pytest 6.2.5
requests 2.26.0
requests-toolbelt 0.9.1
setuptools 41.2.0
shellingham 1.4.0
six 1.15.0
stevedore 2.0.0
toml 0.10.2
tomlkit 0.7.2
urllib3 1.26.7
virtualenv 20.8.1
virtualenv-clone 0.5.4
virtualenvwrapper 4.8.4
webencodings 0.5.1
WARNING: You are using pip version 19.2.3, however version 21.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
- pytest and operating system versions
pytest 6.2.5 and MacOS BigSur 11.6.
The two patches
changelog/9236.bugfix.rstnew file mode 100644index 000000000..9bcf54a35@@ -0,0 +1 @@+Fixed :fixture:`caplog` (and log reporting) intermittently missing records when a stale logging context from a previous test, e.g. a leaked hook generator, was finalized by the garbage collector: it no longer detaches the shared capture handlers or restores the root logger level while another test still relies on them.src/_pytest/logging.pyindex 6e80e6ba3..70091bb4e 100644@@ -19,6 +19,7 @@ import os from pathlib import Path import re from types import TracebackType+from typing import Any from typing import final from typing import Generic from typing import Literal@@ -342,6 +343,20 @@ def pytest_addoption(parser: Parser) -> None: _HandlerType = TypeVar("_HandlerType", bound=logging.Handler) +# Number of active ``catching_logs`` contexts which attached a given handler to+# a given logger. The same handler instance is reused by several (possibly+# overlapping) contexts, e.g. the ``LoggingPlugin`` handlers, so a handler is+# only detached once the last context which attached it exits. Otherwise a+# context exiting out of order -- e.g. a leaked hook generator finalized later+# by the garbage collector -- would detach a handler another active context+# still relies on (#9236).+_handler_attachments: dict[tuple[logging.Logger, logging.Handler], int] = {}++# Active ``catching_logs`` contexts which changed the root logger level, in the+# order they were entered, so out of order exits restore the right level.+_root_level_contexts: list[catching_logs[Any]] = []++ # Not using @contextmanager for performance reasons. class catching_logs(Generic[_HandlerType]): """Context manager that prepares the whole logging machinery properly."""@@ -353,13 +368,28 @@ class catching_logs(Generic[_HandlerType]): self.level = level self.attached_loggers: list[logging.Logger] = [] + def _attach(self, logger: logging.Logger) -> None:+ key = (logger, self.handler)+ count = _handler_attachments.get(key)+ if count is not None:+ # Re-attach in case it was removed behind our back, e.g. by+ # ``logging.basicConfig(force=True)``.+ logger.addHandler(self.handler)+ _handler_attachments[key] = count + 1+ elif self.handler not in logger.handlers:+ logger.addHandler(self.handler)+ _handler_attachments[key] = 1+ else:+ # Attached by someone else, who is responsible for removing it.+ return+ self.attached_loggers.append(logger)+ def __enter__(self) -> _HandlerType: root_logger = logging.getLogger() if self.level is not None: self.handler.setLevel(self.level) # Attach to root logger.- root_logger.addHandler(self.handler)- self.attached_loggers.append(root_logger)+ self._attach(root_logger) # Attach to all non-propagating loggers (won't reach root). # Note that will miss loggers that *become* non-propagating # after the `__enter__`. Not worth the trouble for now.@@ -369,13 +399,13 @@ class catching_logs(Generic[_HandlerType]): and not logger.propagate and logger is not root_logger ):- logger.addHandler(self.handler)- self.attached_loggers.append(logger)+ self._attach(logger) if self.level is not None: # Non-propagating loggers still inherit the level (unless a logger # explicitly set level), so only do this on the root logger. self.orig_level = root_logger.level root_logger.setLevel(min(self.orig_level, self.level))+ _root_level_contexts.append(self) return self.handler def __exit__(@@ -385,10 +415,23 @@ class catching_logs(Generic[_HandlerType]): exc_tb: TracebackType | None, ) -> None: root_logger = logging.getLogger()- if self.level is not None:- root_logger.setLevel(self.orig_level)+ if self in _root_level_contexts:+ index = _root_level_contexts.index(self)+ if index == len(_root_level_contexts) - 1:+ root_logger.setLevel(self.orig_level)+ else:+ # A context entered after this one is still active: hand it+ # our original level to restore once it exits instead.+ _root_level_contexts[index + 1].orig_level = self.orig_level+ del _root_level_contexts[index] for logger in self.attached_loggers:- logger.removeHandler(self.handler)+ key = (logger, self.handler)+ count = _handler_attachments[key] - 1+ if count:+ _handler_attachments[key] = count+ else:+ del _handler_attachments[key]+ logger.removeHandler(self.handler) self.attached_loggers.clear() testing/logging/test_fixture.pyindex 95c0f44b7..ba4c76894 100644@@ -6,6 +6,8 @@ from collections.abc import Iterator import logging from _pytest.logging import caplog_records_key+from _pytest.logging import catching_logs+from _pytest.logging import LogCaptureHandler from _pytest.pytester import Pytester import pytest @@ -542,3 +544,121 @@ def test_log_report_captures_according_to_config_option_upon_failure( ["*Print message*", "*INFO log message*", "*WARNING log message*"] ) assert result.ret == 1+++def test_caplog_survives_gc_of_leaked_runtest_context(pytester: Pytester) -> None:+ """A stale logging hook context finalized by the garbage collector during+ a later test must not detach the handlers or restore the root logger level+ the current test relies on (#9236)."""+ pytester.makeconftest(+ """+ import pytest++ @pytest.hookimpl(wrapper=True)+ def pytest_runtest_call(item):+ if item.name == "test_leak":+ # Simulate a hook generator left suspended (e.g. by a+ # misbehaving hook wrapper), only finalized later by the gc.+ plugin = item.config.pluginmanager.getplugin("logging-plugin")+ cm = plugin._runtest_for(item, "call")+ cm.__enter__()+ cm.cycle = cm+ return (yield)+ """+ )+ pytester.makepyfile(+ """+ import gc+ import logging++ logger = logging.getLogger(__name__)++ def test_leak():+ pass++ def test_caplog_after_gc(caplog):+ gc.collect()+ logger.info("INFO message")+ logger.error("ERROR message")+ assert caplog.messages == ["INFO message", "ERROR message"]+ """+ )+ result = pytester.runpytest("--log-level=INFO")+ result.assert_outcomes(passed=2)+++def test_caplog_reattached_after_removal_with_leaked_context(+ pytester: Pytester,+) -> None:+ """While a stale logging hook context is still alive, a handler removed by+ a test (e.g. via ``logging.basicConfig(force=True)``) is re-attached for+ the following tests (#9236)."""+ pytester.makeconftest(+ """+ import pytest++ @pytest.hookimpl(wrapper=True)+ def pytest_runtest_call(item):+ if item.name == "test_leak":+ plugin = item.config.pluginmanager.getplugin("logging-plugin")+ cm = plugin._runtest_for(item, "call")+ cm.__enter__()+ item.config.stash_leaked_cm = cm+ return (yield)+ """+ )+ pytester.makepyfile(+ """+ import logging++ logger = logging.getLogger(__name__)++ def test_leak():+ pass++ def test_reconfigure():+ logging.basicConfig(force=True)++ def test_caplog_after(caplog):+ logger.warning("hello")+ assert caplog.messages == ["hello"]+ """+ )+ result = pytester.runpytest_subprocess()+ result.assert_outcomes(passed=3)+++def test_catching_logs_out_of_order_exit() -> None:+ """Overlapping catching_logs contexts sharing a handler keep it attached and+ the root logger level lowered until the last one exits (#9236)."""+ root_logger = logging.getLogger()+ orig_level = root_logger.level+ handler = LogCaptureHandler()++ first = catching_logs(handler, level=logging.INFO)+ second = catching_logs(handler, level=logging.DEBUG)+ first.__enter__()+ second.__enter__()+ assert root_logger.level == min(orig_level, logging.DEBUG)++ # The first context exits while the second one is still active.+ first.__exit__(None, None, None)+ assert handler in root_logger.handlers+ assert root_logger.level == min(orig_level, logging.DEBUG)++ second.__exit__(None, None, None)+ assert handler not in root_logger.handlers+ assert root_logger.level == orig_level+++def test_catching_logs_keeps_externally_attached_handler() -> None:+ """catching_logs does not remove a handler it did not attach itself."""+ root_logger = logging.getLogger()+ handler = LogCaptureHandler()+ root_logger.addHandler(handler)+ try:+ with catching_logs(handler):+ assert handler in root_logger.handlers+ assert handler in root_logger.handlers+ finally:+ root_logger.removeHandler(handler)
changelog/9236.packaging.rstnew file mode 100644index 000000000..76de357da@@ -0,0 +1,6 @@+``pluggy>=1.6`` is now required.++Older pluggy versions stopped running the remaining hook wrapper teardowns once an+old-style ``hookwrapper`` raised in its own teardown. The logging plugin's wrapper was+left suspended, and when it was finalised later it detached the handler that+:fixture:`caplog` of a subsequent test was recording through, so that test saw no records.pyproject.tomlindex e7dcef6e0..8e68acdac 100644@@ -51,7 +51,7 @@ dependencies = [ "exceptiongroup>=1; python_version<'3.11'", "iniconfig>=2", "packaging>=24",- "pluggy>=1.5,<2",+ "pluggy>=1.6,<2", "pygments>=2.15", "tomli>=2; python_version<'3.11'", ]testing/logging/test_fixture.pyindex 95c0f44b7..ce6e8a319 100644@@ -502,6 +502,46 @@ def test_captures_despite_exception(pytester: Pytester) -> None: assert result.ret == 1 +def test_caplog_survives_earlier_hookwrapper_raising_in_teardown(+ pytester: Pytester,+) -> None:+ """#9236: pluggy < 1.6 skipped the remaining teardowns once an old-style+ hookwrapper raised in its own, leaving the logging plugin's wrapper+ suspended. Finalising it later detached the handler ``caplog`` of the+ next test was recording through."""+ pytester.makeconftest(+ """+ import pytest++ @pytest.hookimpl(hookwrapper=True)+ def pytest_runtest_call(item):+ outcome = yield+ outcome.get_result()+ """+ )+ pytester.makepyfile(+ """+ import gc+ import logging++ def test_fails():+ assert False++ def test_caplog(caplog):+ gc.collect()+ logging.getLogger().error("TEST")+ assert caplog.messages == ["TEST"]+ """+ )+ # In-process, the HookRecorder keeps the failing call's CallInfo, whose+ # traceback keeps the suspended wrapper reachable: it is never finalised+ # and the test would pass on the affected pluggy versions too.+ result = pytester.runpytest_subprocess(+ "-W", "ignore::pluggy.PluggyTeardownRaisedWarning"+ )+ result.assert_outcomes(passed=1, failed=1)++ def test_log_report_captures_according_to_config_option_upon_failure( pytester: Pytester, ) -> None:uv.lockindex 065a1fdcd..621c4d9eb 100644@@ -554,7 +554,7 @@ requires-dist = [ { name = "numpy", marker = "extra == 'dev'", specifier = ">=1.26" }, { name = "packaging", specifier = ">=24" }, { name = "pexpect", marker = "extra == 'dev'", specifier = ">=4.9" },- { name = "pluggy", specifier = ">=1.5,<2" },+ { name = "pluggy", specifier = ">=1.6,<2" }, { name = "pygments", specifier = ">=2.15" }, { name = "pytest-xdist", marker = "extra == 'dev'", specifier = ">=3.5" }, { name = "pyyaml", marker = "extra == 'dev'" },