Backtestsscrapyrun-20261025-211112#4827
RFC: Allow usage of file descriptors or stdin with runspider
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches make `/dev/fd/N` process-substitution paths and extensionless FIFOs work, and the independent tests confirm that for both. The issue explicitly names a second use case, `-` for stdin, which Patch A implements (reads sys.stdin.buffer, executes as a module named `<stdin>` for tracebacks) and Patch B does not: with B, `runspider -` still hits the `Path('-').exists()` check and fails, as the independent stdin test shows (PASS for A, FAIL-ASSERT for B). On root cause, A distinguishes regular files (still imported with the extension check) from non-regular files (read and exec'd), which is a direct answer to the issue's concern without loosening validation for ordinary files. B instead relaxes the extension check to 'any extension or none' and swaps `import_module` for a bare `SourceFileLoader.exec_module`, which also drops the sys.modules registration the old path had and gives modules names like `63`; it works but is a less targeted change with a small regression surface. On tests, A adds tests for stdin success, stdin no-spider, stdin unable-to-load, stdin traceback filename, /dev/fd pipe, and a named FIFO, with skips only on platforms lacking /dev/fd or mkfifo (the issue's scenarios are POSIX shells, so the skips would not hold there). B adds a single bash process-substitution test. A also documents the new behavior in commands.rst and the command's long description. One uncertainty for A is that its tests rely on `proc` accepting `input=` and `pass_fds=`, which I cannot verify here, but the independent runs show A's production code covers every scenario the issue names, while B leaves the stdin case unaddressed.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
17m 10s
Wall-clock time in the container
Full test suite
No new failures
134 passed · 41 skipped · 175 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
162vs 46 merged
Changed lines, ours vs merged
The issue
View on GitHubIf this check could be improved, more creative uses of runspider would be admitted. For example, using a file descriptor avoids creating a temporary file (following example taken from the home page):
$ pip install scrapy
$ scrapy runspider <(echo <<EOF)
import scrapy
class BlogSpider(scrapy.Spider):
name = 'blogspider'
start_urls = ['https://blog.scrapinghub.com']
def parse(self, response):
for title in response.css('.post-header>h2'):
yield {'title': title.css('a ::text').get()}
for next_page in response.css('a.next-posts-link'):
yield response.follow(next_page, self.parse)
EOF
Currently, this raises an error because the filename ends up being something like /dev/fd/324. Perhaps the extension check could be removed entirely?
Another use case would be to use - for stdin.
The two patches
docs/topics/commands.rstindex 2294f2d23..79eae9f64 100644@@ -640,6 +640,10 @@ runspider Run the spider defined in the given Python file, without requiring a project. +Use ``-`` as the file name to read the spider code from standard input. Pipes,+such as the file descriptors created by shell process substitution, can also+be used, regardless of their name.+ Supported options: the same as :command:`crawl`. Example usage::@@ -647,6 +651,12 @@ Example usage:: $ scrapy runspider myspider.py [ ... spider starts crawling ... ] + $ cat myspider.py | scrapy runspider -+ [ ... spider starts crawling ... ]++ $ scrapy runspider <(curl -s https://example.com/myspider.py)+ [ ... spider starts crawling ... ]+ .. command:: version versionscrapy/commands/runspider.pyindex ebaa60cfe..18bbae885 100644@@ -3,6 +3,7 @@ from __future__ import annotations import sys from importlib import import_module from pathlib import Path+from types import ModuleType from typing import TYPE_CHECKING, Any, ClassVar from scrapy.commands import BaseRunSpiderCommand@@ -13,7 +14,6 @@ from scrapy.utils.spider import iter_spider_classes if TYPE_CHECKING: import argparse from os import PathLike- from types import ModuleType def _import_file(filepath: str | PathLike[str]) -> ModuleType:@@ -29,6 +29,58 @@ def _import_file(filepath: str | PathLike[str]) -> ModuleType: return module +#: Name of the module created when loading spider code from standard input or+#: from a non-regular file, such as a pipe (e.g. ``/dev/fd/63`` from shell+#: process substitution).+_SOURCE_MODULE_NAME = "__scrapy_runspider__"+++def _load_source(source: bytes, filename: str) -> ModuleType:+ """Execute *source* as the code of a new module and return that module.++ *filename* is used in tracebacks and as the module ``__file__``.+ """+ code = compile(source, filename, "exec", dont_inherit=True)+ module = ModuleType(_SOURCE_MODULE_NAME)+ module.__file__ = filename+ # Registered before execution so that code relying on sys.modules lookups+ # of the defining module (e.g. dataclasses, pickling) works.+ sys.modules[_SOURCE_MODULE_NAME] = module+ try:+ exec(code, module.__dict__) # noqa: S102+ except BaseException:+ del sys.modules[_SOURCE_MODULE_NAME]+ raise+ return module+++def _load_spider_module(filename: str) -> ModuleType:+ """Load the module defined in *filename*.++ ``-`` means standard input. Regular files must have a Python source file+ extension and are imported. Other existing files, like pipes or character+ devices, are read and executed.+ """+ path = Path(filename)+ if filename != "-" and not path.exists():+ raise UsageError(f"File not found: {filename}\n")+ try:+ if filename != "-" and path.is_file():+ return _import_file(path)+ # Only errors reading the source are caught here, so that errors+ # raised by the executed code keep their traceback.+ try:+ if filename == "-":+ source, source_name = sys.stdin.buffer.read(), "<stdin>"+ else:+ source, source_name = path.read_bytes(), filename+ except OSError as e:+ raise UsageError(f"Unable to load {filename!r}: {e}\n") from e+ return _load_source(source, source_name)+ except (ImportError, ValueError) as e:+ raise UsageError(f"Unable to load {filename!r}: {e}\n") from e++ class Command(BaseRunSpiderCommand): default_settings: ClassVar[dict[str, Any]] = { "SPIDER_LOADER_CLASS": DummySpiderLoader@@ -41,18 +93,16 @@ class Command(BaseRunSpiderCommand): return "Run a spider from a Python file, no project required" def long_desc(self) -> str:- return "Run the spider defined in the given file"+ return (+ "Run the spider defined in the given file. Use - to read the spider"+ " code from standard input."+ ) def run(self, args: list[str], opts: argparse.Namespace) -> None: if len(args) != 1: raise UsageError- filename = Path(args[0])- if not filename.exists():- raise UsageError(f"File not found: {filename}\n")- try:- module = _import_file(filename)- except (ImportError, ValueError) as e:- raise UsageError(f"Unable to load {str(filename)!r}: {e}\n") from e+ filename = args[0]+ module = _load_spider_module(filename) spclasses = list(iter_spider_classes(module)) if not spclasses: raise UsageError(f"No spider found in file: {filename}\n")tests/test_command_runspider.pyindex 113b655ed..ac076b8c7 100644@@ -2,7 +2,9 @@ from __future__ import annotations import asyncio import inspect+import os import platform+import subprocess import sys from typing import TYPE_CHECKING @@ -155,11 +157,93 @@ class MySpider(scrapy.Spider): log = self.get_log(tmp_path, "", name="myspider.txt") assert "Unable to load" in log + def test_runspider_module_os_error_traceback(self, tmp_path: Path) -> None:+ code = "open('missing-config.json')\n"+ log = self.get_log(tmp_path, code, name="oserrorspider.py")+ assert "Unable to load" not in log+ assert "oserrorspider.py" in log, log+ assert "missing-config.json" in log, log+ def test_start_errors(self, tmp_path: Path) -> None: log = self.get_log(tmp_path, self.badspider, name="badspider.py") assert "start" in log assert "badspider.py" in log, log + def test_runspider_stdin(self, tmp_path: Path) -> None:+ _, _, log = proc("runspider", "-", cwd=tmp_path, input=self.debug_log_spider)+ assert "DEBUG: It Works!" in log+ assert "INFO: Spider closed (finished)" in log++ def test_runspider_stdin_no_spider_found(self, tmp_path: Path) -> None:+ returncode, _, log = proc(+ "runspider",+ "-",+ cwd=tmp_path,+ input="from scrapy.spiders import Spider\n",+ )+ assert returncode != 0+ assert "No spider found in file: -" in log++ def test_runspider_stdin_unable_to_load(self, tmp_path: Path) -> None:+ returncode, _, log = proc(+ "runspider",+ "-",+ cwd=tmp_path,+ input="import scrapy_runspider_missing_module\n",+ )+ assert returncode != 0+ assert "Unable to load '-'" in log++ def test_runspider_stdin_start_errors(self, tmp_path: Path) -> None:+ _, _, log = proc("runspider", "-", cwd=tmp_path, input=self.badspider)+ assert "oops!" in log+ assert "<stdin>" in log, log++ @pytest.mark.skipif(+ not os.path.isdir("/dev/fd"), reason="/dev/fd required for file descriptors"+ )+ def test_runspider_file_descriptor(self, tmp_path: Path) -> None:+ # Equivalent to: scrapy runspider <(echo "...")+ read_fd, write_fd = os.pipe()+ try:+ with os.fdopen(write_fd, "w", encoding="utf-8") as f:+ f.write(self.debug_log_spider)+ _, _, log = proc(+ "runspider", f"/dev/fd/{read_fd}", cwd=tmp_path, pass_fds=(read_fd,)+ )+ finally:+ os.close(read_fd)+ assert "DEBUG: It Works!" in log+ assert "INFO: Spider closed (finished)" in log++ @pytest.mark.skipif(+ not hasattr(os, "mkfifo"), reason="named pipes (FIFOs) required"+ )+ def test_runspider_named_pipe(self, tmp_path: Path) -> None:+ fifo = tmp_path / "spider_pipe"+ os.mkfifo(fifo)+ # Opening a FIFO for writing blocks until a reader opens it, so the+ # writer runs in a child process started before the command.+ writer = subprocess.Popen(+ [+ sys.executable,+ "-c",+ "import sys; open(sys.argv[1], 'w', encoding='utf-8')"+ ".write(sys.argv[2])",+ str(fifo),+ self.debug_log_spider,+ ]+ )+ try:+ _, _, log = proc("runspider", str(fifo), cwd=tmp_path)+ finally:+ # The writer stays blocked if the command never opened the FIFO.+ if writer.poll() is None:+ writer.kill()+ writer.wait()+ assert "DEBUG: It Works!" in log+ assert "INFO: Spider closed (finished)" in log+ def test_errorcount_exit_code(self, tmp_path: Path) -> None: spider_code = """ import scrapy
scrapy/commands/runspider.pyindex ebaa60cfe..d07a789f5 100644@@ -1,7 +1,8 @@ from __future__ import annotations -import sys-from importlib import import_module+import importlib.util+import os+from importlib.machinery import SourceFileLoader from pathlib import Path from typing import TYPE_CHECKING, Any, ClassVar @@ -17,15 +18,19 @@ if TYPE_CHECKING: def _import_file(filepath: str | PathLike[str]) -> ModuleType:- abspath = Path(filepath).resolve()- if abspath.suffix not in {".py", ".pyw"}:+ # os.path.abspath(), not Path.resolve(), to avoid following symlinks:+ # fd paths such as /dev/fd/63 resolve to an unopenable pipe: target.+ abspath = Path(os.path.abspath(filepath)) # noqa: PTH100+ if abspath.suffix and abspath.suffix not in {".py", ".pyw"}: raise ValueError(f"Not a Python source file: {abspath}")- dirname = str(abspath.parent)- sys.path = [dirname, *sys.path]- try:- module = import_module(abspath.stem)- finally:- sys.path.pop(0)+ module_name = abspath.stem or "spidermodule"+ # An explicit loader is required: without it extensionless file paths, such+ # as the /dev/fd paths of process substitution, get no loader assigned.+ loader = SourceFileLoader(module_name, str(abspath))+ spec = importlib.util.spec_from_file_location(module_name, abspath, loader=loader)+ assert spec+ module = importlib.util.module_from_spec(spec)+ loader.exec_module(module) return module tests/test_command_runspider.pyindex 113b655ed..46be5cdcf 100644@@ -3,11 +3,13 @@ from __future__ import annotations import asyncio import inspect import platform+import subprocess import sys from typing import TYPE_CHECKING import pytest +from scrapy.utils.test import get_testenv from tests.spiders import ExceptionSpider, NoRequestsSpider from tests.utils.cmdline import proc @@ -155,6 +157,25 @@ class MySpider(scrapy.Spider): log = self.get_log(tmp_path, "", name="myspider.txt") assert "Unable to load" in log + @pytest.mark.skipif(+ platform.system() == "Windows", reason="process substitution requires bash"+ )+ def test_runspider_fd(self) -> None:+ command = (+ f"{sys.executable} -m scrapy.cmdline runspider"+ f" <(cat <<'EOF'\n{self.debug_log_spider}\nEOF\n)"+ )+ p = subprocess.run(+ ["bash", "-c", command],+ check=False,+ capture_output=True,+ encoding="utf-8",+ timeout=15,+ env=get_testenv(),+ )+ assert p.returncode == 0, p.stderr+ assert "DEBUG: It Works!" in p.stderr+ def test_start_errors(self, tmp_path: Path) -> None: log = self.get_log(tmp_path, self.badspider, name="badspider.py") assert "start" in log