Backtestsscrapyrun-20261025-211112#7796
get_func_args_dict doesn't support PEP649 lazy annotation loading
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the reported problem the same way: a `_signature` helper that passes `annotation_format=Format.FORWARDREF` on 3.14+, used in `get_func_args_dict` and in `_warn_spider_arg`. All five independent tests go from NameError on base to PASS on both, so correctness on the issue's scenarios is equal. The difference is in tests and scope. Patch A adds a small reusable helper (`tests/utils/pep649.py`) and exercises the fix at several levels: unit tests for `get_func_args_dict`/`get_func_args` with the issue's exact middleware shape, `_warn_spider_arg`, a real downloader-middleware run through the middleware manager (the user's actual use case), and a stats-collector regression. Patch B adds two narrower unit tests built around inline exec/compile. Both sets would fail before and pass after, and both are correctly skipped only below 3.14, where the bug cannot occur, so neither skip undermines the evidence. Patch A also rewrites `get_spec` on 3.14+ to avoid `getfullargspec`, which is beyond the issue's ask; it is carefully done (keeps bound `self`, ignores `__wrapped__`, filters to positional kinds, maps failures to TypeError like the stdlib) and gated to 3.14, so it adds modest risk but does not obscure the core fix. Patch B is tighter and has a nice touch documenting that `annotation` may now be a ForwardRef, but that craft edge does not outweigh A's materially stronger, scenario-level test coverage. Close call; A wins on tests, which rank above scope.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
32m 21s
Wall-clock time in the container
Full test suite
No new failures
1,124 passed · 35 skipped · 1,162 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
205vs 75 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
Hi scrapy team, in 3.14 inspect.signature() on type-annotated functions isn't safe at runtime as types are lazy in 3.14.
Steps to Reproduce
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import scrapy
class SomeMiddleware:
def process_exception(self, request: scrapy.Request, exception: Exception) -> None:
pass
from scrapy.utils.python import get_func_args_dict
get_func_args_dict(SomeMiddleware().process_exception) # crashes here
Expected behavior:get_func_args_dict returns (dict of arguments)
Actual behavior:
Raises NameError: name 'scrapy' is not defined instead of returning.
Reproduces how often:
N/A / Always with Python 3.14
Versions
Scrapy : 2.17.0
lxml : 6.1.1
libxml2 : 2.14.6
cssselect : 1.5.0
parsel : 1.11.0
w3lib : 2.4.1
Twisted : 26.4.0
Python : 3.14.6 (main, Jun 23 2026, 15:46:31) [Clang 22.1.3 ]
pyOpenSSL : 26.3.0 (OpenSSL 4.0.1 9 Jun 2026)
cryptography : 49.0.0
Platform : macOS-26.5.2-arm64-arm-64bit-Mach-O
Additional context
Found this when using ruff's TC rules (TC002).
Potential fix for annotation_format=inspect.Format.FORWARDREF passed to inspect but not familiar enough with the codebase to say whether that's correct or not / what you do with the values downstream.
The two patches
scrapy/utils/decorators.pyindex 4960dc27a..2924c81f9 100644@@ -10,6 +10,7 @@ from twisted.internet.defer import Deferred, maybeDeferred from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.utils.asyncio import run_in_thread from scrapy.utils.defer import deferred_from_coro+from scrapy.utils.python import _signature if TYPE_CHECKING: from collections.abc import AsyncGenerator, Callable, Coroutine@@ -109,7 +110,7 @@ def _warn_spider_arg( ): """Decorator to warn if a ``spider`` argument is passed to a function.""" - sig = inspect.signature(func)+ sig = _signature(func) def check_args(*args: _P.args, **kwargs: _P.kwargs) -> None: bound = sig.bind(*args, **kwargs)scrapy/utils/python.pyindex 8a7517c1d..371a83523 100644@@ -178,6 +178,19 @@ def binary_is_text(data: bytes) -> bool: return all(c not in _BINARYCHARS for c in data) +def _signature(func: Callable[..., Any], **kwargs: Any) -> inspect.Signature:+ """Return :func:`inspect.signature` of *func* without evaluating its+ annotations.++ On Python 3.14+ (:pep:`649`) annotations are evaluated lazily, so names+ only imported under :data:`typing.TYPE_CHECKING` would make the default+ evaluation raise :exc:`NameError`.+ """+ if sys.version_info >= (3, 14):+ kwargs["annotation_format"] = inspect.Format.FORWARDREF+ return inspect.signature(func, **kwargs)++ def get_func_args_dict( func: Callable[..., Any], stripself: bool = False ) -> Mapping[str, inspect.Parameter]:@@ -190,7 +203,7 @@ def get_func_args_dict( args: Mapping[str, inspect.Parameter] try:- sig = inspect.signature(func)+ sig = _signature(func) except ValueError: return {} @@ -241,17 +254,46 @@ def get_spec(func: Callable[..., Any]) -> tuple[list[str], dict[str, Any]]: """ if inspect.isfunction(func) or inspect.ismethod(func):- spec = inspect.getfullargspec(func)+ target = func elif hasattr(func, "__call__"): # noqa: B004- spec = inspect.getfullargspec(func.__call__)+ target = func.__call__ else: raise TypeError(f"{type(func)} is not callable") - defaults: tuple[Any, ...] = spec.defaults or ()+ spec_args: list[str]+ defaults: tuple[Any, ...]+ if sys.version_info >= (3, 14) and (+ inspect.isfunction(target) or inspect.ismethod(target)+ ):+ # inspect.getfullargspec() evaluates annotations, which can fail+ # with lazy annotations (PEP 649), so build the same data from a+ # signature that doesn't evaluate them. Like getfullargspec(), keep+ # the bound argument of methods and don't follow __wrapped__.+ if inspect.ismethod(target):+ target = target.__func__+ try:+ sig = _signature(target, follow_wrapped=False)+ except Exception as ex:+ raise TypeError("unsupported callable") from ex+ spec_args = []+ defaults = ()+ for param in sig.parameters.values():+ if param.kind not in (+ inspect.Parameter.POSITIONAL_ONLY,+ inspect.Parameter.POSITIONAL_OR_KEYWORD,+ ):+ continue+ spec_args.append(param.name)+ if param.default is not param.empty:+ defaults += (param.default,)+ else:+ spec = inspect.getfullargspec(target)+ spec_args = spec.args+ defaults = spec.defaults or () - firstdefault = len(spec.args) - len(defaults)- args = spec.args[:firstdefault]- kwargs = dict(zip(spec.args[firstdefault:], defaults, strict=False))+ firstdefault = len(spec_args) - len(defaults)+ args = spec_args[:firstdefault]+ kwargs = dict(zip(spec_args[firstdefault:], defaults, strict=False)) return args, kwargs tests/test_downloadermiddleware.pyindex 92eb18d33..926251660 100644@@ -17,6 +17,7 @@ from scrapy.utils.defer import maybe_deferred_to_future from scrapy.utils.python import to_bytes from scrapy.utils.test import get_crawler, get_from_asyncio_queue from tests.utils.decorators import coroutine_test+from tests.utils.pep649 import define_lazy, requires_lazy_annotations if TYPE_CHECKING: from collections.abc import AsyncGenerator@@ -346,3 +347,36 @@ class TestDeprecatedSpiderArg(TestManagerBase): result = await mwman.download_async(download_func, req) assert result is resp assert not download_func.called+++class TestLazyAnnotations(TestManagerBase):+ @requires_lazy_annotations+ @coroutine_test+ async def test_lazy_annotations(self):+ req = Request("http://example.com/index.html")+ resp = Response("http://example.com/index.html")++ mw_cls = define_lazy(+ """+ class LazyAnnotationsMiddleware:+ def process_request(self, request: scrapy.Request) -> None:+ 1 / 0++ def process_response(+ self, request: scrapy.Request, response: scrapy.Response+ ) -> scrapy.Response:+ return response++ def process_exception(+ self, request: scrapy.Request, exception: Exception+ ) -> scrapy.Response:+ return resp+ """,+ "LazyAnnotationsMiddleware",+ resp=resp,+ )++ async with self.get_mwman() as mwman:+ mwman._add_middleware(mw_cls())+ ret = await self._download(mwman, req)+ assert ret is resptests/test_stats.pyindex c33a00090..c9cd438f4 100644@@ -13,6 +13,7 @@ from scrapy.statscollectors import DummyStatsCollector, StatsCollector from scrapy.utils.test import get_crawler from tests.spiders import SimpleSpider from tests.utils.decorators import coroutine_test+from tests.utils.pep649 import define_lazy, requires_lazy_annotations if TYPE_CHECKING: from scrapy.crawler import Crawler@@ -133,6 +134,27 @@ class TestStatsCollector: ): assert stats.get_stats(spider) == {"test": "value"} + @requires_lazy_annotations+ def test_deprecated_spider_arg_lazy_annotations(self, crawler: Crawler) -> None:+ cls = define_lazy(+ """+ class CustomStatsCollector(StatsCollector):+ def set_value(+ self, key: str, value: Any, spider: scrapy.Spider | None = None+ ) -> None:+ super().set_value(key, value)+ """,+ "CustomStatsCollector",+ StatsCollector=StatsCollector,+ )+ stats = cls(crawler)+ with pytest.warns(+ ScrapyDeprecationWarning,+ match=r"Passing a 'spider' argument to .*CustomStatsCollector\.set_value\(\) is deprecated",+ ):+ stats.set_value("test", "value", spider=None)+ assert stats.get_value("test") == "value"+ @coroutine_test async def test_deprecated_spider_arg_custom_collector(self) -> None: # the class reimplements many methods because those are called during the test crawltests/test_utils_decorators.pyindex 807294a57..bf7d9caa6 100644@@ -10,6 +10,7 @@ from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.utils.decorators import _warn_spider_arg, deprecated, inthread from scrapy.utils.defer import maybe_deferred_to_future from tests.utils.decorators import coroutine_test+from tests.utils.pep649 import define_lazy, requires_lazy_annotations if TYPE_CHECKING: from collections.abc import AsyncGenerator@@ -111,3 +112,20 @@ class TestWarnSpiderArg: results = [item async for item in parse("response", spider="spider")] assert results == ["response"]++ @requires_lazy_annotations+ def test_lazy_annotations(self):+ parse = _warn_spider_arg(+ define_lazy(+ """+ def parse(response: scrapy.Response, spider: Spider = None) -> Item:+ return response+ """,+ "parse",+ )+ )++ with pytest.warns(+ ScrapyDeprecationWarning, match=r"Passing a 'spider' argument"+ ):+ assert parse("response", spider="spider") == "response"tests/test_utils_python.pyindex c3b5dfc99..8d941ffbd 100644@@ -14,12 +14,15 @@ from scrapy.utils.python import ( MutableAsyncChain, binary_is_text, get_func_args,+ get_func_args_dict,+ get_spec, memoizemethod_noargs, to_bytes, to_unicode, without_none_values, ) from tests.utils.decorators import coroutine_test+from tests.utils.pep649 import define_lazy, requires_lazy_annotations if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterable, Mapping@@ -205,3 +208,47 @@ def test_without_none_values( value: Mapping[_KT, _VT] | Iterable[_KT], expected: dict[_KT, _VT] | Iterable[_KT] ) -> None: assert without_none_values(value) == expected+++@requires_lazy_annotations+def test_get_func_args_lazy_annotations():+ cls = define_lazy(+ """+ class SomeMiddleware:+ def process_exception(+ self, request: scrapy.Request, exception: Exception+ ) -> None:+ pass+ """,+ "SomeMiddleware",+ )+ args = get_func_args_dict(cls().process_exception)+ assert list(args) == ["request", "exception"]+ assert get_func_args(cls().process_exception) == ["request", "exception"]+ assert get_func_args(cls.process_exception, stripself=True) == [+ "request",+ "exception",+ ]+++@requires_lazy_annotations+def test_get_spec_lazy_annotations():+ cls = define_lazy(+ """+ class CustomRequest:+ def __init__(+ self, url: scrapy.Url, /, callback: Callback, flags: Flags = 0+ ) -> None:+ pass++ def __call__(self, val: Undefined, flags: Undefined = 0) -> None:+ pass+ """,+ "CustomRequest",+ )+ assert get_spec(cls.__init__) == (["self", "url", "callback"], {"flags": 0})+ assert get_spec(cls("u", None).__init__) == (+ ["self", "url", "callback"],+ {"flags": 0},+ )+ assert get_spec(cls("u", None)) == (["self", "val"], {"flags": 0})tests/utils/pep649.pynew file mode 100644index 000000000..2052c3caf@@ -0,0 +1,25 @@+from __future__ import annotations++import sys+import textwrap+from typing import Any++import pytest++requires_lazy_annotations = pytest.mark.skipif(+ sys.version_info < (3, 14), reason="PEP 649 lazy annotations need Python 3.14+"+)+++def define_lazy(source: str, name: str, **namespace: Any) -> Any:+ """Execute *source* without inheriting ``from __future__ import+ annotations`` and return the object called *name* defined in it.++ Annotations in *source* are thus lazily evaluated (PEP 649), which allows+ referencing names that are not defined at runtime, as with imports+ guarded by ``typing.TYPE_CHECKING``. Keyword arguments are made+ available to *source* as globals.+ """+ code = compile(textwrap.dedent(source), "<pep649>", "exec", dont_inherit=True)+ exec(code, namespace) # noqa: S102+ return namespace[name]
scrapy/utils/decorators.pyindex 4960dc27a..2924c81f9 100644@@ -10,6 +10,7 @@ from twisted.internet.defer import Deferred, maybeDeferred from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.utils.asyncio import run_in_thread from scrapy.utils.defer import deferred_from_coro+from scrapy.utils.python import _signature if TYPE_CHECKING: from collections.abc import AsyncGenerator, Callable, Coroutine@@ -109,7 +110,7 @@ def _warn_spider_arg( ): """Decorator to warn if a ``spider`` argument is passed to a function.""" - sig = inspect.signature(func)+ sig = _signature(func) def check_args(*args: _P.args, **kwargs: _P.kwargs) -> None: bound = sig.bind(*args, **kwargs)scrapy/utils/python.pyindex 8a7517c1d..40fc05257 100644@@ -178,11 +178,30 @@ def binary_is_text(data: bytes) -> bool: return all(c not in _BINARYCHARS for c in data) +# PEP 649 (Python 3.14+) made annotation evaluation lazy, so inspect.signature()+# can raise NameError for names imported only under TYPE_CHECKING. We only need+# parameter names, kinds and defaults, so leave such annotations as ForwardRefs.+if sys.version_info >= (3, 14):+ from annotationlib import Format++ def _signature(func: Callable[..., Any]) -> inspect.Signature:+ return inspect.signature(func, annotation_format=Format.FORWARDREF)++else:++ def _signature(func: Callable[..., Any]) -> inspect.Signature:+ return inspect.signature(func)++ def get_func_args_dict( func: Callable[..., Any], stripself: bool = False ) -> Mapping[str, inspect.Parameter]: """Return the argument dict of a callable object. + Annotations are not evaluated, so on Python 3.14 and later the ``annotation``+ attribute of the returned parameters may be a ``ForwardRef`` instead of the+ resolved type.+ .. versionadded:: 2.14 """ if not callable(func):@@ -190,7 +209,7 @@ def get_func_args_dict( args: Mapping[str, inspect.Parameter] try:- sig = inspect.signature(func)+ sig = _signature(func) except ValueError: return {} tests/test_utils_decorators.pyindex 807294a57..4c29d2917 100644@@ -1,7 +1,8 @@ from __future__ import annotations +import sys import warnings-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, Any import pytest from twisted.internet.defer import Deferred@@ -12,7 +13,7 @@ from scrapy.utils.defer import maybe_deferred_to_future from tests.utils.decorators import coroutine_test if TYPE_CHECKING:- from collections.abc import AsyncGenerator+ from collections.abc import AsyncGenerator, Callable class TestDeprecated:@@ -77,6 +78,31 @@ class TestWarnSpiderArg: ): assert parse("response", spider="spider") == "response" + @pytest.mark.skipif(+ sys.version_info < (3, 14),+ reason="annotations are only lazily evaluated since Python 3.14 (PEP 649)",+ )+ def test_sync_warns_with_unresolvable_annotations(self):+ # dont_inherit=True, or the module's future import stringizes the annotations+ namespace: dict[str, Any] = {}+ exec( # pylint: disable=exec-used+ compile(+ "def parse(response: OnlyAtTypeCheckingTime,"+ " spider: OnlyAtTypeCheckingTime | None = None): return response",+ "<test>",+ "exec",+ dont_inherit=True,+ ),+ namespace,+ )+ parse_func: Callable[..., str] = namespace["parse"]+ parse = _warn_spider_arg(parse_func)++ with pytest.warns(+ ScrapyDeprecationWarning, match=r"Passing a 'spider' argument"+ ):+ assert parse("response", spider="spider") == "response"+ def test_sync_no_warning_without_spider_arg(self): @_warn_spider_arg def parse(response: str, spider: str | None = None) -> str:tests/test_utils_python.pyindex c3b5dfc99..099b5ccc2 100644@@ -4,7 +4,7 @@ import functools import operator import platform import sys-from typing import TYPE_CHECKING, TypeVar+from typing import TYPE_CHECKING, Any, TypeVar import pytest @@ -190,6 +190,25 @@ def test_get_func_args(): ] +@pytest.mark.skipif(+ sys.version_info < (3, 14),+ reason="annotations are only lazily evaluated since Python 3.14 (PEP 649)",+)+def test_get_func_args_unresolvable_annotations():+ # dont_inherit=True, or the module's future import stringizes the annotations+ namespace: dict[str, Any] = {}+ exec( # pylint: disable=exec-used+ compile(+ "def f(a: OnlyAtTypeCheckingTime, b: int = 1) -> OnlyAtTypeCheckingTime: pass",+ "<test>",+ "exec",+ dont_inherit=True,+ ),+ namespace,+ )+ assert get_func_args(namespace["f"]) == ["a", "b"]++ @pytest.mark.parametrize( ("value", "expected"), [