Backtestspytestrun-20261106-234554#2781
--strict does not complain when non-existent markers are specified as args to pytest
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the root cause the same way: collect the identifiers from the compiled -m expression and, when strict/strict_markers is on, raise UsageError for any not in the `markers` ini list. Both pass all independent tests. The differences are in scope, tests, and craft. Patch B is tighter: it extracts idents with a single ast.walk over the already-built AST, factors the existing marker-name parsing and strict-check logic into two small helpers in structures.py that MarkGenerator now shares, and uses an error message that mirrors the existing '... not found in `markers` configuration option' wording so the -m failure reads like the decorator failure the issue asks to match. Its tests are considerably more thorough: parametrized over --strict-markers, --strict, ini strict_markers, ini strict, and addopts; over compound expressions, kwargs form, and multiple unknowns (checking sorted output); plus registered/builtin (skip, xfail) and non-strict regression cases. Minor nits: it reads expr._idents from outside the module and gives idents a default of frozenset() in __init__. Patch A reaches further than needed: it adds a RegisteredMarker NamedTuple and _iter_registered_markers to Config, rewrites the `--markers` output path (unrelated behavior with some risk), threads an idents set through Scanner and changes the internal expression() return type. Its unit tests rely on a FakeConfig cast to Config to call a Config method on a non-Config, which is brittle, and its pytester coverage is narrower (no ini-based strict, no kwargs/multiple-unknown cases). Patch B resolves the issue with less churn, a more consistent error, and stronger tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
3m 56s
Wall-clock time in the container
Full test suite
No new failures
2,358 passed · 23 skipped · 2,381 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
147vs 227 merged
Changed lines, ours vs merged
The issue
View on GitHubI'm not seeing pytest complain in strict mode when filtering by markers that I haven't specified.
e.g. pytest --strict -m marker_that_i_tyoped_wrong mytests/
This will complete quite happily, deselecting all tests.
It would be nice if the behaviour were able to be changed, perhaps by another flag (to avoid causing sadness to anyone relying on the current behaviour), such that if the mark expression contained invalid marks it would cause a failure in the same way that a test containing invalid marks would.
The two patches
changelog/7023.improvement.rstnew file mode 100644index 000000000..84fdb2bdc@@ -0,0 +1 @@+When :confval:`strict_markers` (or :confval:`strict`) is enabled, passing an unregistered marker name to ``-m`` now results in a usage error, instead of silently deselecting all tests.doc/en/how-to/mark.rstindex e22219414..6ca1cc500 100644@@ -83,7 +83,8 @@ the warning for custom marks by registering them in your configuration file or using a custom ``pytest_configure`` hook. When the :confval:`strict_markers` configuration option is set, any unknown marks applied-with the ``@pytest.mark.name_of_the_mark`` decorator will trigger an error. You can+with the ``@pytest.mark.name_of_the_mark`` decorator, as well as any unknown marks used+in a ``-m`` mark expression, will trigger an error. You can enforce this validation in your project by setting :confval:`strict_markers` in your configuration: .. tab:: tomlsrc/_pytest/mark/__init__.pyindex 5b6ae8aa8..dd1be5adc 100644@@ -20,6 +20,8 @@ from .structures import MARK_GEN from .structures import MarkDecorator from .structures import MarkGenerator from .structures import ParameterSet+from .structures import registered_marker_names+from .structures import strict_markers_enabled from _pytest.compat import NOTSET from _pytest.config import Config from _pytest.config import ExitCode@@ -264,6 +266,14 @@ def deselect_by_mark(items: list[Item], config: Config) -> None: return expr = _parse_expression(matchexpr, "Wrong expression passed to '-m'")+ if strict_markers_enabled(config):+ unknown = expr._idents - registered_marker_names(config)+ if unknown:+ raise UsageError(+ "Unknown marker(s) passed to '-m': "+ + ", ".join(repr(name) for name in sorted(unknown))+ + " not found in `markers` configuration option"+ ) remaining: list[Item] = [] deselected: list[Item] = [] for item in items:src/_pytest/mark/expression.pyindex 4b4a68d8a..545b1b9dc 100644@@ -314,12 +314,16 @@ class Expression: The expression can be evaluated against different matchers. """ - __slots__ = ("_code", "input")+ __slots__ = ("_code", "_idents", "input") - def __init__(self, input: str, code: types.CodeType) -> None:+ def __init__(+ self, input: str, code: types.CodeType, idents: frozenset[str] = frozenset()+ ) -> None: #: The original input line, as a string. self.input: Final = input self._code: Final = code+ #: The identifiers (e.g. mark names) referenced in the expression.+ self._idents: Final = idents @classmethod def compile(cls, input: str) -> Expression:@@ -335,7 +339,12 @@ class Expression: filename="<pytest match expression>", mode="eval", )- return Expression(input, code)+ idents = frozenset(+ node.id[len(IDENT_PREFIX) :]+ for node in ast.walk(astexpr)+ if isinstance(node, ast.Name)+ )+ return Expression(input, code, idents) def evaluate(self, matcher: ExpressionMatcher) -> bool: """Evaluate the match expression.src/_pytest/mark/structures.pyindex 18057f0b0..b159db51a 100644@@ -563,6 +563,25 @@ if TYPE_CHECKING: ... +def registered_marker_names(config: Config) -> set[str]:+ """Return the names of the markers registered via the ``markers`` ini option."""+ names = set()+ for line in config.getini("markers"):+ # example lines: "skipif(condition): skip the given test if..."+ # or "hypothesis: tests which use Hypothesis", so to get the+ # marker name we split on both `:` and `(`.+ names.add(line.split(":")[0].split("(")[0].strip())+ return names+++def strict_markers_enabled(config: Config) -> bool:+ """Whether unregistered markers should be treated as errors."""+ strict_markers = config.getini("strict_markers")+ if strict_markers is None:+ strict_markers = config.getini("strict")+ return bool(strict_markers)++ @final class MarkGenerator: """Factory for :class:`MarkDecorator` objects - exposed as@@ -604,12 +623,7 @@ class MarkGenerator: # name is in the set we definitely know it, but a mark may be known and # not in the set. We therefore start by updating the set! if name not in self._markers:- for line in self._config.getini("markers"):- # example lines: "skipif(condition): skip the given test if..."- # or "hypothesis: tests which use Hypothesis", so to get the- # marker name we split on both `:` and `(`.- marker = line.split(":")[0].split("(")[0].strip()- self._markers.add(marker)+ self._markers.update(registered_marker_names(self._config)) # If the name is not in the set of known marks after updating, # then it really is time to issue a warning or an error.@@ -619,10 +633,7 @@ class MarkGenerator: __tracebackhide__ = True fail(f"Unknown '{name}' mark, did you mean 'parametrize'?") - strict_markers = self._config.getini("strict_markers")- if strict_markers is None:- strict_markers = self._config.getini("strict")- if strict_markers:+ if strict_markers_enabled(self._config): fail( f"{name!r} not found in `markers` configuration option", pytrace=False,testing/test_mark.pyindex c70376e70..00b2ffb72 100644@@ -219,6 +219,93 @@ def test_strict_prohibits_unregistered_markers(pytester: Pytester, option: str) ) +@pytest.mark.parametrize(+ "option",+ [+ "--strict-markers",+ "--strict",+ "strict_markers = true",+ "strict = true",+ "addopts = --strict-markers",+ ],+)+@pytest.mark.parametrize(+ ("expr", "unknown"),+ [+ ("unregisteredmark", "'unregisteredmark'"),+ ("registered and not unregisteredmark", "'unregisteredmark'"),+ ("registered or unregisteredmark(x=1)", "'unregisteredmark'"),+ ("zzz or registered or aaa", "'aaa', 'zzz'"),+ ],+)+def test_strict_prohibits_unregistered_markers_in_mark_expression(+ pytester: Pytester, option: str, expr: str, unknown: str+) -> None:+ """Unregistered markers in -m are errors in strict mode (#7023)."""+ pytester.makepyfile(+ """+ import pytest+ @pytest.mark.registered+ def test_hello():+ pass+ """+ )+ ini_lines = ["[pytest]", "markers = registered"]+ args = ["-m", expr]+ if option.startswith("-"):+ args.insert(0, option)+ else:+ ini_lines.append(option)+ pytester.makeini("\n".join(ini_lines))+ result = pytester.runpytest(*args)+ assert result.ret == ExitCode.USAGE_ERROR+ result.stderr.fnmatch_lines(+ [+ f"ERROR: Unknown marker(s) passed to '-m': {unknown} "+ "not found in `markers` configuration option"+ ]+ )+ result.stdout.no_fnmatch_line("*passed*")+++@pytest.mark.parametrize(+ "expr", ["registered", "not registered", "registered(x=1)", "skip or xfail"]+)+def test_strict_allows_registered_markers_in_mark_expression(+ pytester: Pytester, expr: str+) -> None:+ pytester.makepyfile(+ """+ import pytest+ @pytest.mark.registered+ def test_hello():+ pass+ """+ )+ pytester.makeini(+ """+ [pytest]+ markers = registered+ """+ )+ result = pytester.runpytest("--strict-markers", "-m", expr)+ assert result.ret in (ExitCode.OK, ExitCode.NO_TESTS_COLLECTED)+++def test_non_strict_allows_unregistered_markers_in_mark_expression(+ pytester: Pytester,+) -> None:+ pytester.makepyfile(+ """+ def test_hello():+ pass+ """+ )+ result = pytester.runpytest("-m", "unregisteredmark")+ assert result.ret == ExitCode.NO_TESTS_COLLECTED+ result.stdout.fnmatch_lines(["*1 deselected*"])++ @pytest.mark.parametrize( ("expr", "expected_passed"), [
changelog/2781.feature.rstnew file mode 100644index 000000000..d3a30efc2@@ -0,0 +1 @@+When :confval:`strict_markers` is enabled, marker names used in :option:`-m` expressions are now validated against registered markers.doc/en/reference/reference.rstindex 262782462..362d5c917 100644@@ -2479,6 +2479,9 @@ passed multiple times. The expected format is ``name=value``. For example:: If set to ``true``, markers not registered in the ``markers`` section of the configuration file will raise errors. + This applies both to markers applied to tests (e.g. ``@pytest.mark.slow``) and to marker+ names used in :option:`-m` expressions.+ .. tab:: toml .. code-block:: tomlsrc/_pytest/config/__init__.pyindex c1a80ed16..7a7573de0 100644@@ -36,6 +36,7 @@ from typing import Final from typing import final from typing import IO from typing import Literal+from typing import NamedTuple from typing import TextIO from typing import TYPE_CHECKING import warnings@@ -1044,6 +1045,17 @@ class _DeprecatedInicfgProxy(MutableMapping[str, Any]): return len(self._config._inicfg) +class RegisteredMarker(NamedTuple):+ """A marker registered in the configuration."""++ #: The marker name (e.g., ``skipif``).+ name: str+ #: The full marker signature (e.g., ``skipif(condition)``).+ signature: str+ #: The marker description.+ description: str++ @final class Config: """Access to configuration values, pluginmanager and plugin hooks.@@ -1788,6 +1800,18 @@ class Config: self._inicache[canonical_name] = val = self._getini(canonical_name) return val + def _iter_registered_markers(self) -> Iterator[RegisteredMarker]:+ """Iterate over all markers registered in the configuration."""+ for line in self.getini("markers"):+ # Example lines: "skipif(condition): skip the given test if..."+ # or "hypothesis: tests which use Hypothesis", so to get the+ # marker name we split on both `:` and `(`.+ parts = line.split(":", 1)+ signature = parts[0]+ description = parts[1].strip() if len(parts) == 2 else ""+ name = signature.split("(")[0].strip()+ yield RegisteredMarker(name, signature, description)+ # Meant for easy monkeypatching by legacypath plugin. # Can be inlined back (with no cover removed) once legacypath is gone. def _getini_unknown_type(self, name: str, type: str, value: object):src/_pytest/mark/__init__.pyindex 5b6ae8aa8..73354506d 100644@@ -140,12 +140,9 @@ def pytest_cmdline_main(config: Config) -> int | ExitCode | None: if config.option.markers: config._do_configure() tw = _pytest.config.create_terminal_writer(config)- for line in config.getini("markers"):- parts = line.split(":", 1)- name = parts[0]- rest = parts[1] if len(parts) == 2 else ""- tw.write(f"@pytest.mark.{name}:", bold=True)- tw.line(rest)+ for marker in config._iter_registered_markers():+ tw.write(f"@pytest.mark.{marker.signature}:", bold=True)+ tw.line(f" {marker.description}" if marker.description else "") tw.line() config._ensure_unconfigure() return 0@@ -264,6 +261,8 @@ def deselect_by_mark(items: list[Item], config: Config) -> None: return expr = _parse_expression(matchexpr, "Wrong expression passed to '-m'")+ _validate_marker_names(expr, config)+ remaining: list[Item] = [] deselected: list[Item] = [] for item in items:@@ -276,6 +275,28 @@ def deselect_by_mark(items: list[Item], config: Config) -> None: items[:] = remaining +def _validate_marker_names(expr: Expression, config: Config) -> None:+ """Validate that all marker names in the expression are registered.++ Only validates when strict_markers is enabled.+ """+ strict_markers = config.getini("strict_markers")+ if strict_markers is None:+ strict_markers = config.getini("strict")+ if not strict_markers:+ return++ registered_markers = {m.name for m in config._iter_registered_markers()}++ unknown_markers = expr.idents() - registered_markers+ if unknown_markers:+ unknown_str = ", ".join(sorted(unknown_markers))+ raise UsageError(+ f"Unknown marker(s) in '-m' expression: {unknown_str}. "+ "Use 'pytest --markers' to see available markers."+ )++ def _parse_expression(expr: str, exc_message: str) -> Expression: try: return Expression.compile(expr)src/_pytest/mark/expression.pyindex 4b4a68d8a..65a776202 100644@@ -70,10 +70,11 @@ class Token: class Scanner:- __slots__ = ("current", "input", "tokens")+ __slots__ = ("current", "idents", "input", "tokens") def __init__(self, input: str) -> None: self.input = input+ self.idents: set[str] = set() self.tokens = self.lex(input) self.current = next(self.tokens) @@ -163,13 +164,13 @@ class Scanner: IDENT_PREFIX = "$" -def expression(s: Scanner) -> ast.Expression:+def expression(s: Scanner) -> tuple[ast.Expression, frozenset[str]]: if s.accept(TokenType.EOF): ret: ast.expr = ast.Constant(False) else: ret = expr(s) s.accept(TokenType.EOF, reject=True)- return ast.fix_missing_locations(ast.Expression(ret))+ return ast.fix_missing_locations(ast.Expression(ret)), frozenset(s.idents) def expr(s: Scanner) -> ast.expr:@@ -197,6 +198,7 @@ def not_expr(s: Scanner) -> ast.expr: return ret ident = s.accept(TokenType.IDENT) if ident:+ s.idents.add(ident.value) name = ast.Name(IDENT_PREFIX + ident.value, ast.Load()) if s.accept(TokenType.LPAREN): ret = ast.Call(func=name, args=[], keywords=all_kwargs(s))@@ -314,12 +316,16 @@ class Expression: The expression can be evaluated against different matchers. """ - __slots__ = ("_code", "input")+ __slots__ = ("_code", "_idents", "input") - def __init__(self, input: str, code: types.CodeType) -> None:+ def __init__(+ self, input: str, code: types.CodeType, idents: frozenset[str]+ ) -> None: #: The original input line, as a string. self.input: Final = input self._code: Final = code+ #: All identifiers which appear in the expression.+ self._idents: Final = idents @classmethod def compile(cls, input: str) -> Expression:@@ -329,13 +335,17 @@ class Expression: :raises SyntaxError: If the expression is malformed. """- astexpr = expression(Scanner(input))+ astexpr, idents = expression(Scanner(input)) code = compile( astexpr, filename="<pytest match expression>", mode="eval", )- return Expression(input, code)+ return Expression(input, code, idents)++ def idents(self) -> frozenset[str]:+ """Return the set of all identifiers which appear in the expression."""+ return self._idents def evaluate(self, matcher: ExpressionMatcher) -> bool: """Evaluate the match expression.src/_pytest/mark/structures.pyindex 18057f0b0..635cf9d47 100644@@ -604,12 +604,9 @@ class MarkGenerator: # name is in the set we definitely know it, but a mark may be known and # not in the set. We therefore start by updating the set! if name not in self._markers:- for line in self._config.getini("markers"):- # example lines: "skipif(condition): skip the given test if..."- # or "hypothesis: tests which use Hypothesis", so to get the- # marker name we split on both `:` and `(`.- marker = line.split(":")[0].split("(")[0].strip()- self._markers.add(marker)+ self._markers.update(+ m.name for m in self._config._iter_registered_markers()+ ) # If the name is not in the set of known marks after updating, # then it really is time to issue a warning or an error.testing/test_mark.pyindex c70376e70..2e105dfe4 100644@@ -1,12 +1,19 @@ # mypy: allow-untyped-defs from __future__ import annotations +from collections.abc import Iterator import os import sys+from typing import cast from unittest import mock +from _pytest.config import Config from _pytest.config import ExitCode+from _pytest.config import RegisteredMarker+from _pytest.config import UsageError+from _pytest.mark import _validate_marker_names from _pytest.mark import MarkGenerator+from _pytest.mark.expression import Expression from _pytest.mark.structures import _EmptyParameterSetMark from _pytest.mark.structures import EMPTY_PARAMETERSET_OPTION from _pytest.nodes import Collector@@ -219,6 +226,114 @@ def test_strict_prohibits_unregistered_markers(pytester: Pytester, option: str) ) +class TestValidateMarkerNames:+ """Tests for _validate_marker_names (issue #2781)."""++ class FakeConfig:+ def __init__(+ self,+ markers: list[str],+ strict_markers: bool | None = None,+ strict: bool = False,+ ) -> None:+ self._ini: dict[str, list[str] | bool | None] = {+ "markers": markers,+ "strict_markers": strict_markers,+ "strict": strict,+ }++ def getini(self, name: str) -> list[str] | bool | None:+ return self._ini[name]++ def _iter_registered_markers(self) -> Iterator[RegisteredMarker]:+ yield from Config._iter_registered_markers(cast(Config, self))++ def _make_config(+ self,+ strict_markers: bool | None = None,+ strict: bool = False,+ ) -> Config:+ return cast(+ Config,+ self.FakeConfig(+ markers=["registered: a registered marker"],+ strict_markers=strict_markers,+ strict=strict,+ ),+ )++ def test_unknown_marker_with_strict_markers(self) -> None:+ expr = Expression.compile("unknown_marker")++ with pytest.raises(UsageError, match=r"Unknown marker.*unknown_marker"):+ _validate_marker_names(expr, self._make_config(strict_markers=True))++ def test_unknown_marker_with_strict(self) -> None:+ expr = Expression.compile("unknown_marker")++ with pytest.raises(UsageError, match=r"Unknown marker.*unknown_marker"):+ _validate_marker_names(expr, self._make_config(strict=True))++ def test_registered_marker_passes(self) -> None:+ expr = Expression.compile("registered")++ _validate_marker_names(expr, self._make_config(strict_markers=True))++ def test_no_validation_without_strict(self) -> None:+ expr = Expression.compile("any_marker")++ _validate_marker_names(expr, self._make_config())+++@pytest.fixture+def markexpr_pytester(pytester: Pytester) -> Pytester:+ pytester.makeini(+ """+ [pytest]+ markers =+ registered: a registered marker+ """+ )+ pytester.makepyfile(+ """+ import pytest++ @pytest.mark.registered+ def test_registered():+ pass++ def test_plain():+ pass+ """+ )+ return pytester+++@pytest.mark.parametrize("option", ["--strict-markers", "--strict"])+def test_strict_prohibits_unregistered_markers_in_markexpr(+ markexpr_pytester: Pytester, option: str+) -> None:+ result = markexpr_pytester.runpytest(option, "-m", "registered or unregisteredmark")+ assert result.ret == ExitCode.USAGE_ERROR+ result.stderr.fnmatch_lines(+ ["*Unknown marker(s) in '-m' expression: unregisteredmark*"]+ )+++def test_strict_allows_registered_markers_in_markexpr(+ markexpr_pytester: Pytester,+) -> None:+ result = markexpr_pytester.runpytest("--strict-markers", "-m", "registered")+ result.assert_outcomes(passed=1, deselected=1)+++def test_unregistered_markers_in_markexpr_allowed_without_strict(+ markexpr_pytester: Pytester,+) -> None:+ result = markexpr_pytester.runpytest("-m", "unregisteredmark")+ result.assert_outcomes(deselected=2)++ @pytest.mark.parametrize( ("expr", "expected_passed"), [testing/test_mark_expression.pyindex 3a606bac1..ac9c882e6 100644@@ -336,3 +336,21 @@ def test_str_keyword_expressions( expr: str, expected: bool, mark_matcher: MarkMatcher ) -> None: assert evaluate(expr, mark_matcher) is expected+++@pytest.mark.parametrize(+ ("expr", "expected_idents"),+ (+ ("", frozenset()),+ ("foo", frozenset(["foo"])),+ ("foo and bar", frozenset(["foo", "bar"])),+ ("foo or bar", frozenset(["foo", "bar"])),+ ("not foo", frozenset(["foo"])),+ ("(foo and bar) or baz", frozenset(["foo", "bar", "baz"])),+ ("foo and foo", frozenset(["foo"])), # Duplicates are deduplicated.+ ("mark(a=1)", frozenset(["mark"])), # Only marker name, not kwargs.+ ),+)+def test_expression_idents(expr: str, expected_idents: frozenset[str]) -> None:+ """Test that Expression.idents() returns the identifiers in the expression."""+ assert Expression.compile(expr).idents() == expected_idents