Backtestsscrapyrun-20261025-211112#4827

Issue #4827 Merged fix PR #8036 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

RFC: Allow usage of file descriptors or stdin with runspider

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches make `/dev/fd/N` process-substitution paths and extensionless FIFOs work, and the independent tests confirm that for both. The issue explicitly names a second use case, `-` for stdin, which Patch A implements (reads sys.stdin.buffer, executes as a module named `<stdin>` for tracebacks) and Patch B does not: with B, `runspider -` still hits the `Path('-').exists()` check and fails, as the independent stdin test shows (PASS for A, FAIL-ASSERT for B). On root cause, A distinguishes regular files (still imported with the extension check) from non-regular files (read and exec'd), which is a direct answer to the issue's concern without loosening validation for ordinary files. B instead relaxes the extension check to 'any extension or none' and swaps `import_module` for a bare `SourceFileLoader.exec_module`, which also drops the sys.modules registration the old path had and gives modules names like `63`; it works but is a less targeted change with a small regression surface. On tests, A adds tests for stdin success, stdin no-spider, stdin unable-to-load, stdin traceback filename, /dev/fd pipe, and a named FIFO, with skips only on platforms lacking /dev/fd or mkfifo (the issue's scenarios are POSIX shells, so the skips would not hold there). B adds a single bash process-substitution test. A also documents the new behavior in commands.rst and the command's long description. One uncertainty for A is that its tests rely on `proc` accepting `input=` and `pass_fds=`, which I cannot verify here, but the independent runs show A's production code covers every scenario the issue names, while B leaves the stdin case unaddressed.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

17m 10s

Wall-clock time in the container

Full test suite

No new failures

134 passed · 41 skipped · 175 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

162vs 46 merged

Changed lines, ours vs merged

https://github.com/scrapy/scrapy/blob/004b40a7193a7c0c5138abe764cad6d1d9a4fc57/scrapy/commands/runspider.py#L14

If this check could be improved, more creative uses of runspider would be admitted. For example, using a file descriptor avoids creating a temporary file (following example taken from the home page):

$ pip install scrapy
$ scrapy runspider <(echo <<EOF)
import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)
EOF

Currently, this raises an error because the filename ends up being something like /dev/fd/324. Perhaps the extension check could be removed entirely?

Another use case would be to use - for stdin.

The two patches

Our patch+153 −9
docs/topics/commands.rstindex 2294f2d23..79eae9f64 100644@@ -640,6 +640,10 @@ runspider  Run the spider defined in the given Python file, without requiring a project. +Use ``-`` as the file name to read the spider code from standard input. Pipes,+such as the file descriptors created by shell process substitution, can also+be used, regardless of their name.+ Supported options: the same as :command:`crawl`.  Example usage::@@ -647,6 +651,12 @@ Example usage::     $ scrapy runspider myspider.py     [ ... spider starts crawling ... ] +    $ cat myspider.py | scrapy runspider -+    [ ... spider starts crawling ... ]++    $ scrapy runspider <(curl -s https://example.com/myspider.py)+    [ ... spider starts crawling ... ]+ .. command:: version  versionscrapy/commands/runspider.pyindex ebaa60cfe..18bbae885 100644@@ -3,6 +3,7 @@ from __future__ import annotations import sys from importlib import import_module from pathlib import Path+from types import ModuleType from typing import TYPE_CHECKING, Any, ClassVar  from scrapy.commands import BaseRunSpiderCommand@@ -13,7 +14,6 @@ from scrapy.utils.spider import iter_spider_classes if TYPE_CHECKING:     import argparse     from os import PathLike-    from types import ModuleType   def _import_file(filepath: str | PathLike[str]) -> ModuleType:@@ -29,6 +29,58 @@ def _import_file(filepath: str | PathLike[str]) -> ModuleType:     return module  +#: Name of the module created when loading spider code from standard input or+#: from a non-regular file, such as a pipe (e.g. ``/dev/fd/63`` from shell+#: process substitution).+_SOURCE_MODULE_NAME = "__scrapy_runspider__"+++def _load_source(source: bytes, filename: str) -> ModuleType:+    """Execute *source* as the code of a new module and return that module.++    *filename* is used in tracebacks and as the module ``__file__``.+    """+    code = compile(source, filename, "exec", dont_inherit=True)+    module = ModuleType(_SOURCE_MODULE_NAME)+    module.__file__ = filename+    # Registered before execution so that code relying on sys.modules lookups+    # of the defining module (e.g. dataclasses, pickling) works.+    sys.modules[_SOURCE_MODULE_NAME] = module+    try:+        exec(code, module.__dict__)  # noqa: S102+    except BaseException:+        del sys.modules[_SOURCE_MODULE_NAME]+        raise+    return module+++def _load_spider_module(filename: str) -> ModuleType:+    """Load the module defined in *filename*.++    ``-`` means standard input. Regular files must have a Python source file+    extension and are imported. Other existing files, like pipes or character+    devices, are read and executed.+    """+    path = Path(filename)+    if filename != "-" and not path.exists():+        raise UsageError(f"File not found: {filename}\n")+    try:+        if filename != "-" and path.is_file():+            return _import_file(path)+        # Only errors reading the source are caught here, so that errors+        # raised by the executed code keep their traceback.+        try:+            if filename == "-":+                source, source_name = sys.stdin.buffer.read(), "<stdin>"+            else:+                source, source_name = path.read_bytes(), filename+        except OSError as e:+            raise UsageError(f"Unable to load {filename!r}: {e}\n") from e+        return _load_source(source, source_name)+    except (ImportError, ValueError) as e:+        raise UsageError(f"Unable to load {filename!r}: {e}\n") from e++ class Command(BaseRunSpiderCommand):     default_settings: ClassVar[dict[str, Any]] = {         "SPIDER_LOADER_CLASS": DummySpiderLoader@@ -41,18 +93,16 @@ class Command(BaseRunSpiderCommand):         return "Run a spider from a Python file, no project required"      def long_desc(self) -> str:-        return "Run the spider defined in the given file"+        return (+            "Run the spider defined in the given file. Use - to read the spider"+            " code from standard input."+        )      def run(self, args: list[str], opts: argparse.Namespace) -> None:         if len(args) != 1:             raise UsageError-        filename = Path(args[0])-        if not filename.exists():-            raise UsageError(f"File not found: {filename}\n")-        try:-            module = _import_file(filename)-        except (ImportError, ValueError) as e:-            raise UsageError(f"Unable to load {str(filename)!r}: {e}\n") from e+        filename = args[0]+        module = _load_spider_module(filename)         spclasses = list(iter_spider_classes(module))         if not spclasses:             raise UsageError(f"No spider found in file: {filename}\n")tests/test_command_runspider.pyindex 113b655ed..ac076b8c7 100644@@ -2,7 +2,9 @@ from __future__ import annotations  import asyncio import inspect+import os import platform+import subprocess import sys from typing import TYPE_CHECKING @@ -155,11 +157,93 @@ class MySpider(scrapy.Spider):         log = self.get_log(tmp_path, "", name="myspider.txt")         assert "Unable to load" in log +    def test_runspider_module_os_error_traceback(self, tmp_path: Path) -> None:+        code = "open('missing-config.json')\n"+        log = self.get_log(tmp_path, code, name="oserrorspider.py")+        assert "Unable to load" not in log+        assert "oserrorspider.py" in log, log+        assert "missing-config.json" in log, log+     def test_start_errors(self, tmp_path: Path) -> None:         log = self.get_log(tmp_path, self.badspider, name="badspider.py")         assert "start" in log         assert "badspider.py" in log, log +    def test_runspider_stdin(self, tmp_path: Path) -> None:+        _, _, log = proc("runspider", "-", cwd=tmp_path, input=self.debug_log_spider)+        assert "DEBUG: It Works!" in log+        assert "INFO: Spider closed (finished)" in log++    def test_runspider_stdin_no_spider_found(self, tmp_path: Path) -> None:+        returncode, _, log = proc(+            "runspider",+            "-",+            cwd=tmp_path,+            input="from scrapy.spiders import Spider\n",+        )+        assert returncode != 0+        assert "No spider found in file: -" in log++    def test_runspider_stdin_unable_to_load(self, tmp_path: Path) -> None:+        returncode, _, log = proc(+            "runspider",+            "-",+            cwd=tmp_path,+            input="import scrapy_runspider_missing_module\n",+        )+        assert returncode != 0+        assert "Unable to load '-'" in log++    def test_runspider_stdin_start_errors(self, tmp_path: Path) -> None:+        _, _, log = proc("runspider", "-", cwd=tmp_path, input=self.badspider)+        assert "oops!" in log+        assert "<stdin>" in log, log++    @pytest.mark.skipif(+        not os.path.isdir("/dev/fd"), reason="/dev/fd required for file descriptors"+    )+    def test_runspider_file_descriptor(self, tmp_path: Path) -> None:+        # Equivalent to: scrapy runspider <(echo "...")+        read_fd, write_fd = os.pipe()+        try:+            with os.fdopen(write_fd, "w", encoding="utf-8") as f:+                f.write(self.debug_log_spider)+            _, _, log = proc(+                "runspider", f"/dev/fd/{read_fd}", cwd=tmp_path, pass_fds=(read_fd,)+            )+        finally:+            os.close(read_fd)+        assert "DEBUG: It Works!" in log+        assert "INFO: Spider closed (finished)" in log++    @pytest.mark.skipif(+        not hasattr(os, "mkfifo"), reason="named pipes (FIFOs) required"+    )+    def test_runspider_named_pipe(self, tmp_path: Path) -> None:+        fifo = tmp_path / "spider_pipe"+        os.mkfifo(fifo)+        # Opening a FIFO for writing blocks until a reader opens it, so the+        # writer runs in a child process started before the command.+        writer = subprocess.Popen(+            [+                sys.executable,+                "-c",+                "import sys; open(sys.argv[1], 'w', encoding='utf-8')"+                ".write(sys.argv[2])",+                str(fifo),+                self.debug_log_spider,+            ]+        )+        try:+            _, _, log = proc("runspider", str(fifo), cwd=tmp_path)+        finally:+            # The writer stays blocked if the command never opened the FIFO.+            if writer.poll() is None:+                writer.kill()+            writer.wait()+        assert "DEBUG: It Works!" in log+        assert "INFO: Spider closed (finished)" in log+     def test_errorcount_exit_code(self, tmp_path: Path) -> None:         spider_code = """ import scrapy 
Merged fix · PR #8036+36 −10
scrapy/commands/runspider.pyindex ebaa60cfe..d07a789f5 100644@@ -1,7 +1,8 @@ from __future__ import annotations -import sys-from importlib import import_module+import importlib.util+import os+from importlib.machinery import SourceFileLoader from pathlib import Path from typing import TYPE_CHECKING, Any, ClassVar @@ -17,15 +18,19 @@ if TYPE_CHECKING:   def _import_file(filepath: str | PathLike[str]) -> ModuleType:-    abspath = Path(filepath).resolve()-    if abspath.suffix not in {".py", ".pyw"}:+    # os.path.abspath(), not Path.resolve(), to avoid following symlinks:+    # fd paths such as /dev/fd/63 resolve to an unopenable pipe: target.+    abspath = Path(os.path.abspath(filepath))  # noqa: PTH100+    if abspath.suffix and abspath.suffix not in {".py", ".pyw"}:         raise ValueError(f"Not a Python source file: {abspath}")-    dirname = str(abspath.parent)-    sys.path = [dirname, *sys.path]-    try:-        module = import_module(abspath.stem)-    finally:-        sys.path.pop(0)+    module_name = abspath.stem or "spidermodule"+    # An explicit loader is required: without it extensionless file paths, such+    # as the /dev/fd paths of process substitution, get no loader assigned.+    loader = SourceFileLoader(module_name, str(abspath))+    spec = importlib.util.spec_from_file_location(module_name, abspath, loader=loader)+    assert spec+    module = importlib.util.module_from_spec(spec)+    loader.exec_module(module)     return module  tests/test_command_runspider.pyindex 113b655ed..46be5cdcf 100644@@ -3,11 +3,13 @@ from __future__ import annotations import asyncio import inspect import platform+import subprocess import sys from typing import TYPE_CHECKING  import pytest +from scrapy.utils.test import get_testenv from tests.spiders import ExceptionSpider, NoRequestsSpider from tests.utils.cmdline import proc @@ -155,6 +157,25 @@ class MySpider(scrapy.Spider):         log = self.get_log(tmp_path, "", name="myspider.txt")         assert "Unable to load" in log +    @pytest.mark.skipif(+        platform.system() == "Windows", reason="process substitution requires bash"+    )+    def test_runspider_fd(self) -> None:+        command = (+            f"{sys.executable} -m scrapy.cmdline runspider"+            f" <(cat <<'EOF'\n{self.debug_log_spider}\nEOF\n)"+        )+        p = subprocess.run(+            ["bash", "-c", command],+            check=False,+            capture_output=True,+            encoding="utf-8",+            timeout=15,+            env=get_testenv(),+        )+        assert p.returncode == 0, p.stderr+        assert "DEBUG: It Works!" in p.stderr+     def test_start_errors(self, tmp_path: Path) -> None:         log = self.get_log(tmp_path, self.badspider, name="badspider.py")         assert "start" in log