Backtestsscrapyrun-20261025-211112#6989

Issue #6989 Merged fix PR #7832 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Proposal: CLI improvements

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

The issue's concrete complaints are (1) `scrapy runspider spider3` with a name from `scrapy list` just says 'File not found' with no guidance, (2) crawl/runspider are not clearly distinguished in help/docs, and (3) running naively yields no stored output, which surprises users. Patch A addresses all three: `runspider` now detects when the argument is a project spider name (guarded by `inside_project()`, WARN_ONLY, and a broad except so the hint can never mask the original error) and tells the user to run `scrapy crawl <name>`; `crawl`, `check`, `fetch`, `shell`, and `parse` route unknown names through a shared `_load_spider` that raises a UsageError with a context-sensitive hint (use runspider for a file, `scrapy list` inside a project, 'no active project' otherwise) instead of a KeyError traceback; and both `crawl` and `runspider` gain `long_desc` text plus docs explaining that items are not stored unless `-o/-O` or FEEDS is used. The change to `crawl` (loading the class then passing it to `crawler_process.crawl`) is behaviour-preserving since `crawl()` already accepts a class. The `parse` change from log-and-continue to UsageError is a reasonable tightening. Patch A also adds a substantial test class covering unknown names across commands, the file-path-to-crawl hint, outside-project behaviour, the runspider-to-crawl hint, broken spider modules not hiding the error, over-long names, and help text content; these would fail on base and are not tautological. Patch B only reworded two `short_desc` strings and a few doc sentences; it adds no tests and leaves the actual error-message scenario from the issue unchanged (runspider still just says 'File not found'), and its docs edit oddly replaces the `crawl` line in the command list example with `runspider`. Independent tests confirm this: A passes all five, B fails the runspider-hint test and both help-text tests. A's extra scope (shared loader helper across commands) is in service of the same confusion the issue describes and adds little risk given the defensive guards. A is clearly the better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

20m 16s

Wall-clock time in the container

Full test suite

No new failures

280 passed · 30 skipped · 314 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

235vs 15 merged

Changed lines, ours vs merged

Summary

  • Improve consistency in CLI arguments.
  • Modernize the usage.
  • Possibly implement introduce a new, higher level command with its own sub-commands so as not to break existing code.

Motivation

I find the CLI a little awkward and asymmetric: I'm always a little surprised about what's required to get expected results although I've used Scrapy for years.

Here's one example that I just experienced:

Now, let me see what spiders are available.

$ scrapy list

spider1
spider2
spider3

Ah hah. I want to run spider3. Let's see how to do it:

$ scrapy

Scrapy 2.13.3 - active project: my_project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  check         Check spider contracts
  crawl         Run a spider
  edit          Edit spider
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  list          List available spiders
  parse         Parse URL (using its spider) and print the results
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

Use "scrapy <command> -h" to see more info about a command

Ok, runspider - Run a self-contained spider (without creating a project) rings a bell.

And I definitely don't want to create a new project, so that sounds right.

$ scrapy runspider spider3

Usage
=====
  scrapy runspider [options] <spider_file>
scrapy: error: File not found: spider3

Ok, I need to refer to it by filename...

$ scrapy runspider path/to/spider3

Hmm. No output at the command line (stdout) nor to a file. (??) Let me look at the CLI help...

Etc. There are several other scenarios I can think of. Another is that crawl and runspider could be documented to more clearly distinguish them.

Describe alternatives you've considered

Writing a wrapper script using a modern Python CLI library.

Additional context

I've written a couple of dozen Scrapy Spiders, but these things still trip me up every time. Especially how running crawl or runspider naively with no options is basically a fail.

Is there interest in a set of Issues to address various opportunities for improvements and PRs for them?

The two patches

Our patch+214 −21
docs/topics/commands.rstindex e8a843e80..88a5cf5e0 100644@@ -114,7 +114,7 @@ some usage help and the available commands::       scrapy <command> [options] [args]      Available commands:-      crawl         Run a spider+      crawl         Run a spider of the current project       fetch         Fetch a URL using the Scrapy downloader     [...] @@ -263,7 +263,14 @@ crawl * Syntax: ``scrapy crawl <spider>`` * Requires project: *yes* -Start crawling using a spider.+Start crawling using the spider of the current project with the given name.+Use :command:`list` to see the names of available spiders, and+:command:`runspider` to run a spider defined in a standalone Python file+instead.++Scraped items are not stored anywhere by default: use ``-o``/``-O`` (e.g.+``-o items.json``, or ``-o -:jsonl`` to print them to the standard output) or+the :setting:`FEEDS` setting to store them.  Supported options: @@ -572,13 +579,20 @@ runspider * Requires project: *no*  Run a spider self-contained in a Python file, without having to create a-project.+project. The argument is a path to a Python file, not a spider name; to run a+spider of the current project by name, use :command:`crawl` instead.++As with :command:`crawl`, scraped items are not stored anywhere by default:+use ``-o``/``-O`` or the :setting:`FEEDS` setting to store them.  Example usage::      $ scrapy runspider myspider.py     [ ... spider starts crawling ... ] +    $ scrapy runspider -o items.json myspider.py+    [ ... spider starts crawling and appends the result to items.json ... ]+ .. command:: version  versionscrapy/commands/__init__.pyindex d9c919db9..7cf5d989f 100644@@ -17,13 +17,44 @@ from twisted.python import failure from scrapy.exceptions import ScrapyDeprecationWarning, UsageError from scrapy.utils.conf import arglist_to_dict, feed_process_params_from_cli from scrapy.utils.deprecate import method_is_overridden+from scrapy.utils.project import inside_project from scrapy.utils.python import global_object_name  if TYPE_CHECKING:     from collections.abc import Iterable +    from scrapy import Spider     from scrapy.crawler import Crawler, CrawlerProcessBase     from scrapy.settings import Settings+    from scrapy.spiderloader import SpiderLoaderProtocol+++def _looks_like_spider_file(arg: str) -> bool:+    path = Path(arg)+    if path.suffix in {".py", ".pyw"}:+        return True+    try:+        return path.is_file()+    except (OSError, ValueError):  # e.g. a name too long to be a path+        return False+++def _load_spider(spider_loader: SpiderLoaderProtocol, name: str) -> type[Spider]:+    """Return the spider class named *name*, raising :exc:`UsageError` with+    a hint about how to proceed if there is no such spider."""+    try:+        return spider_loader.load(name)+    except KeyError:+        if _looks_like_spider_file(name):+            hint = f"To run a spider defined in a file, use: scrapy runspider {name}"+        elif inside_project():+            hint = 'Use "scrapy list" to see available spiders.'+        else:+            hint = (+                "No active project: spiders can only be referenced by name "+                "from within a project."+            )+        raise UsageError(f"Spider not found: {name}\n{hint}\n") from None   class ScrapyCommand(ABC):scrapy/commands/check.pyindex 2113c19d2..01cf4f7af 100644@@ -7,7 +7,7 @@ from unittest import TextTestResult as _TextTestResult from unittest import TextTestRunner  from scrapy import Spider-from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _load_spider from scrapy.contracts import ContractsManager from scrapy.utils.conf import build_component_list from scrapy.utils.misc import load_object, set_environ@@ -92,7 +92,7 @@ class Command(ScrapyCommand):          with set_environ(SCRAPY_CHECK="true"):             for spidername in args or spider_loader.list():-                spidercls = spider_loader.load(spidername)+                spidercls = _load_spider(spider_loader, spidername)                 spidercls.start = start  # type: ignore[method-assign]                  tested_methods = conman.tested_methods_from_spidercls(spidercls)scrapy/commands/crawl.pyindex 866ba9f6b..62470a98e 100644@@ -2,7 +2,7 @@ from __future__ import annotations  from typing import TYPE_CHECKING -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import BaseRunSpiderCommand, _load_spider from scrapy.exceptions import UsageError  if TYPE_CHECKING:@@ -16,7 +16,17 @@ class Command(BaseRunSpiderCommand):         return "[options] <spider>"      def short_desc(self) -> str:-        return "Run a spider"+        return "Run a spider of the current project"++    def long_desc(self) -> str:+        return (+            "Run a spider of the current project, given its name. "+            'Use "scrapy list" to see available spiders, and '+            '"scrapy runspider" to run a spider defined in a standalone file. '+            "Scraped items are not stored anywhere unless you use -o/-O "+            "(e.g. -o items.json, or -o -:jsonl for standard output) or "+            "configure the FEEDS setting."+        )      def run(self, args: list[str], opts: argparse.Namespace) -> None:         if len(args) < 1:@@ -28,7 +38,8 @@ class Command(BaseRunSpiderCommand):         spname = args[0]          assert self.crawler_process-        self.crawler_process.crawl(spname, **opts.spargs)+        spidercls = _load_spider(self.crawler_process.spider_loader, spname)+        self.crawler_process.crawl(spidercls, **opts.spargs)         self.crawler_process.start()         if self.crawler_process.bootstrap_failed:             self.exitcode = 1scrapy/commands/fetch.pyindex 0b8311efb..35a19cc3c 100644@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, Any  from w3lib.url import is_url -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _load_spider from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils.datatypes import SequenceExclude@@ -86,7 +86,7 @@ class Command(ScrapyCommand):         assert self.crawler_process         spider_loader = self.crawler_process.spider_loader         if opts.spider:-            spidercls = spider_loader.load(opts.spider)+            spidercls = _load_spider(spider_loader, opts.spider)         else:             spidercls = spidercls_for_request(spider_loader, request, spidercls) scrapy/commands/parse.pyindex 93194ded7..f74dbd0b5 100644@@ -10,7 +10,7 @@ from itemadapter import ItemAdapter from twisted.internet.defer import Deferred, maybeDeferred from w3lib.url import is_url -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import BaseRunSpiderCommand, _load_spider from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils import display@@ -246,12 +246,7 @@ class Command(BaseRunSpiderCommand):         assert self.crawler_process         spider_loader = self.crawler_process.spider_loader         if opts.spider:-            try:-                self.spidercls = spider_loader.load(opts.spider)-            except KeyError:-                logger.error(-                    "Unable to find spider: %(spider)s", {"spider": opts.spider}-                )+            self.spidercls = _load_spider(spider_loader, opts.spider)         else:             self.spidercls = spidercls_for_request(spider_loader, Request(url))             if not self.spidercls:scrapy/commands/runspider.pyindex 0b9036457..50003cc50 100644@@ -1,13 +1,16 @@ from __future__ import annotations  import sys+import warnings from importlib import import_module from pathlib import Path from typing import TYPE_CHECKING, Any, ClassVar  from scrapy.commands import BaseRunSpiderCommand from scrapy.exceptions import UsageError-from scrapy.spiderloader import DummySpiderLoader+from scrapy.spiderloader import DummySpiderLoader, SpiderLoader, get_spider_loader+from scrapy.utils.misc import load_object+from scrapy.utils.project import inside_project from scrapy.utils.spider import iter_spider_classes  if TYPE_CHECKING:@@ -41,13 +44,44 @@ class Command(BaseRunSpiderCommand):         return "Run a self-contained spider (without creating a project)"      def long_desc(self) -> str:-        return "Run the spider defined in the given file"+        return (+            "Run the spider defined in the given Python file, without needing "+            "a project. To run a spider of the current project by name, use "+            '"scrapy crawl" instead. Scraped items are not stored anywhere '+            "unless you use -o/-O (e.g. -o items.json, or -o -:jsonl for "+            "standard output) or configure the FEEDS setting."+        )++    def _project_has_spider(self, name: str) -> bool:+        """Return whether *name* is the name of a spider of the active+        project, if any."""+        if not inside_project():+            return False+        assert self.settings is not None+        settings = self.settings.copy()+        # Spider modules that fail to import must not hide the original error.+        settings.set("SPIDER_LOADER_WARN_ONLY", True, priority="cmdline")+        if load_object(settings["SPIDER_LOADER_CLASS"]) is DummySpiderLoader:+            settings.set("SPIDER_LOADER_CLASS", SpiderLoader, priority="cmdline")+        try:+            with warnings.catch_warnings():+                warnings.simplefilter("ignore")+                spider_names = get_spider_loader(settings).list()+        except Exception:  # only a hint, it must not hide the original error+            return False+        return name in spider_names      def run(self, args: list[str], opts: argparse.Namespace) -> None:         if len(args) != 1:             raise UsageError         filename = Path(args[0])         if not filename.exists():+            if self._project_has_spider(args[0]):+                raise UsageError(+                    f"File not found: {filename}\n"+                    f"{args[0]!r} is a spider of the current project, run it "+                    f"with: scrapy crawl {args[0]}\n"+                )             raise UsageError(f"File not found: {filename}\n")         try:             module = _import_file(filename)scrapy/commands/shell.pyindex 19138ffd0..0bde97d80 100644@@ -10,7 +10,7 @@ import asyncio from threading import Thread from typing import TYPE_CHECKING, Any, ClassVar -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _load_spider from scrapy.crawler import AsyncCrawlerProcess, Crawler from scrapy.http import Request from scrapy.shell import Shell@@ -75,7 +75,7 @@ class Command(ScrapyCommand):          spidercls: type[Spider] = DefaultSpider         if opts.spider:-            spidercls = spider_loader.load(opts.spider)+            spidercls = _load_spider(spider_loader, opts.spider)         elif url:             spidercls = spidercls_for_request(                 spider_loader, Request(url), spidercls, log_multiple=Truetests/test_commands.pyindex 3e687e811..006590ceb 100644@@ -665,3 +665,111 @@ class TestPopCommandName:         command = _pop_command_name(argv)         assert command == "crawl"         assert argv == ["scrapy", "-h"]+++class TestSpiderNotFound(TestProjectBase):+    """Commands that take a spider name report unknown spiders as usage+    errors with a hint, instead of a traceback."""++    spider_code = """+import scrapy++class MySpider(scrapy.Spider):+    name = 'myspider'+"""++    def _write_spider(self, proj_path: Path) -> Path:+        path = proj_path / self.project_name / "spiders" / "myspider.py"+        path.write_text(self.spider_code, encoding="utf-8")+        return path++    @pytest.mark.parametrize(+        "args",+        [+            ("crawl", "nonexistent"),+            ("check", "nonexistent"),+            ("fetch", "--spider", "nonexistent", "http://localhost:1/"),+            ("shell", "--spider", "nonexistent", "http://localhost:1/"),+            ("parse", "--spider", "nonexistent", "http://localhost:1/"),+        ],+    )+    def test_unknown_spider_name(+        self, proj_path: Path, args: tuple[str, ...]+    ) -> None:+        self._write_spider(proj_path)+        returncode, _, err = proc(*args, cwd=proj_path)+        assert returncode == 2+        assert "Spider not found: nonexistent" in err+        assert 'Use "scrapy list" to see available spiders.' in err+        assert "Traceback" not in err++    def test_crawl_spider_file(self, proj_path: Path) -> None:+        spider_path = self._write_spider(proj_path)+        rel_path = str(spider_path.relative_to(proj_path))+        returncode, _, err = proc("crawl", rel_path, cwd=proj_path)+        assert returncode == 2+        assert f"Spider not found: {rel_path}" in err+        assert (+            f"To run a spider defined in a file, use: scrapy runspider {rel_path}"+            in err+        )+        assert "Traceback" not in err++    def test_fetch_unknown_spider_outside_project(self, tmp_path: Path) -> None:+        returncode, _, err = proc(+            "fetch", "--spider", "nonexistent", "http://localhost:1/", cwd=tmp_path+        )+        assert returncode == 2+        assert "Spider not found: nonexistent" in err+        assert "No active project" in err+        assert "scrapy list" not in err+        assert "Traceback" not in err++    def test_runspider_project_spider_name(self, proj_path: Path) -> None:+        self._write_spider(proj_path)+        returncode, _, err = proc("runspider", "myspider", cwd=proj_path)+        assert returncode == 2+        assert "File not found: myspider" in err+        assert "run it with: scrapy crawl myspider" in err++    def test_runspider_unknown_name_in_project(self, proj_path: Path) -> None:+        self._write_spider(proj_path)+        returncode, _, err = proc("runspider", "nonexistent", cwd=proj_path)+        assert returncode == 2+        assert "File not found: nonexistent" in err+        assert "scrapy crawl" not in err++    @pytest.mark.parametrize(+        "broken_code",+        [+            "import nonexistent_module_for_test\n",+            "raise RuntimeError('broken spider module')\n",+        ],+    )+    def test_runspider_project_spider_name_broken_module(+        self, proj_path: Path, broken_code: str+    ) -> None:+        self._write_spider(proj_path)+        (proj_path / self.project_name / "spiders" / "broken.py").write_text(+            broken_code, encoding="utf-8"+        )+        returncode, _, err = proc("runspider", "myspider", cwd=proj_path)+        assert returncode == 2+        assert "File not found: myspider" in err+        assert "Traceback" not in err++    def test_crawl_unknown_spider_long_name(self, proj_path: Path) -> None:+        self._write_spider(proj_path)+        name = "a" * 300+        returncode, _, err = proc("crawl", name, cwd=proj_path)+        assert returncode == 2+        assert f"Spider not found: {name}" in err+        assert "Traceback" not in err++    def test_help_distinguishes_crawl_and_runspider(self, proj_path: Path) -> None:+        _, crawl_help, _ = proc("crawl", "-h", cwd=proj_path)+        assert "scrapy runspider" in crawl_help+        assert "-o" in crawl_help+        _, runspider_help, _ = proc("runspider", "-h", cwd=proj_path)+        assert "scrapy crawl" in runspider_help+        assert "-o" in runspider_help 
Merged fix · PR #7832+9 −6
docs/topics/commands.rstindex e8a843e80..343193627 100644@@ -114,8 +114,8 @@ some usage help and the available commands::       scrapy <command> [options] [args]      Available commands:-      crawl         Run a spider       fetch         Fetch a URL using the Scrapy downloader+      runspider     Run a spider from a Python file, no project required     [...]  The first line will print the currently active project if you're inside a@@ -263,7 +263,9 @@ crawl * Syntax: ``scrapy crawl <spider>`` * Requires project: *yes* -Start crawling using a spider.+Start crawling using the spider with the given :attr:`~scrapy.Spider.name`,+which must be one of those that :command:`list` reports. To run a spider from a+file instead, use :command:`runspider`.  Supported options: @@ -571,8 +573,9 @@ runspider * Syntax: ``scrapy runspider <spider_file.py>`` * Requires project: *no* -Run a spider self-contained in a Python file, without having to create a-project.+Run the spider defined in the given Python file, without requiring a project.++Supported options: the same as :command:`crawl`.  Example usage:: scrapy/commands/crawl.pyindex 866ba9f6b..4e086e057 100644@@ -16,7 +16,7 @@ class Command(BaseRunSpiderCommand):         return "[options] <spider>"      def short_desc(self) -> str:-        return "Run a spider"+        return "Run a spider of the current project, by name"      def run(self, args: list[str], opts: argparse.Namespace) -> None:         if len(args) < 1:scrapy/commands/runspider.pyindex 0b9036457..9cdb393ab 100644@@ -38,7 +38,7 @@ class Command(BaseRunSpiderCommand):         return "[options] <spider_file>"      def short_desc(self) -> str:-        return "Run a self-contained spider (without creating a project)"+        return "Run a spider from a Python file, no project required"      def long_desc(self) -> str:         return "Run the spider defined in the given file"