Backtestsscrapyrun-20261025-211112#6989
Proposal: CLI improvements
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The issue's concrete complaints are (1) `scrapy runspider spider3` with a name from `scrapy list` just says 'File not found' with no guidance, (2) crawl/runspider are not clearly distinguished in help/docs, and (3) running naively yields no stored output, which surprises users. Patch A addresses all three: `runspider` now detects when the argument is a project spider name (guarded by `inside_project()`, WARN_ONLY, and a broad except so the hint can never mask the original error) and tells the user to run `scrapy crawl <name>`; `crawl`, `check`, `fetch`, `shell`, and `parse` route unknown names through a shared `_load_spider` that raises a UsageError with a context-sensitive hint (use runspider for a file, `scrapy list` inside a project, 'no active project' otherwise) instead of a KeyError traceback; and both `crawl` and `runspider` gain `long_desc` text plus docs explaining that items are not stored unless `-o/-O` or FEEDS is used. The change to `crawl` (loading the class then passing it to `crawler_process.crawl`) is behaviour-preserving since `crawl()` already accepts a class. The `parse` change from log-and-continue to UsageError is a reasonable tightening. Patch A also adds a substantial test class covering unknown names across commands, the file-path-to-crawl hint, outside-project behaviour, the runspider-to-crawl hint, broken spider modules not hiding the error, over-long names, and help text content; these would fail on base and are not tautological. Patch B only reworded two `short_desc` strings and a few doc sentences; it adds no tests and leaves the actual error-message scenario from the issue unchanged (runspider still just says 'File not found'), and its docs edit oddly replaces the `crawl` line in the command list example with `runspider`. Independent tests confirm this: A passes all five, B fails the runspider-hint test and both help-text tests. A's extra scope (shared loader helper across commands) is in service of the same confusion the issue describes and adds little risk given the defensive guards. A is clearly the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
20m 16s
Wall-clock time in the container
Full test suite
No new failures
280 passed · 30 skipped · 314 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
235vs 15 merged
Changed lines, ours vs merged
The issue
View on GitHubSummary
- Improve consistency in CLI arguments.
- Modernize the usage.
- Possibly implement introduce a new, higher level command with its own sub-commands so as not to break existing code.
Motivation
I find the CLI a little awkward and asymmetric: I'm always a little surprised about what's required to get expected results although I've used Scrapy for years.
Here's one example that I just experienced:
Now, let me see what spiders are available.
$ scrapy list
spider1
spider2
spider3
Ah hah. I want to run spider3. Let's see how to do it:
$ scrapy
Scrapy 2.13.3 - active project: my_project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
check Check spider contracts
crawl Run a spider
edit Edit spider
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
list List available spiders
parse Parse URL (using its spider) and print the results
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
Use "scrapy <command> -h" to see more info about a command
Ok,
runspider - Run a self-contained spider (without creating a project)rings a bell.
And I definitely don't want to create a new project, so that sounds right.
$ scrapy runspider spider3
Usage
=====
scrapy runspider [options] <spider_file>
scrapy: error: File not found: spider3
Ok, I need to refer to it by filename...
$ scrapy runspider path/to/spider3
Hmm. No output at the command line (stdout) nor to a file. (??) Let me look at the CLI help...
Etc. There are several other scenarios I can think of. Another is that crawl and runspider could be documented to more clearly distinguish them.
Describe alternatives you've considered
Writing a wrapper script using a modern Python CLI library.
Additional context
I've written a couple of dozen Scrapy Spiders, but these things still trip me up every time. Especially how running crawl or runspider naively with no options is basically a fail.
Is there interest in a set of Issues to address various opportunities for improvements and PRs for them?
The two patches
docs/topics/commands.rstindex e8a843e80..88a5cf5e0 100644@@ -114,7 +114,7 @@ some usage help and the available commands:: scrapy <command> [options] [args] Available commands:- crawl Run a spider+ crawl Run a spider of the current project fetch Fetch a URL using the Scrapy downloader [...] @@ -263,7 +263,14 @@ crawl * Syntax: ``scrapy crawl <spider>`` * Requires project: *yes* -Start crawling using a spider.+Start crawling using the spider of the current project with the given name.+Use :command:`list` to see the names of available spiders, and+:command:`runspider` to run a spider defined in a standalone Python file+instead.++Scraped items are not stored anywhere by default: use ``-o``/``-O`` (e.g.+``-o items.json``, or ``-o -:jsonl`` to print them to the standard output) or+the :setting:`FEEDS` setting to store them. Supported options: @@ -572,13 +579,20 @@ runspider * Requires project: *no* Run a spider self-contained in a Python file, without having to create a-project.+project. The argument is a path to a Python file, not a spider name; to run a+spider of the current project by name, use :command:`crawl` instead.++As with :command:`crawl`, scraped items are not stored anywhere by default:+use ``-o``/``-O`` or the :setting:`FEEDS` setting to store them. Example usage:: $ scrapy runspider myspider.py [ ... spider starts crawling ... ] + $ scrapy runspider -o items.json myspider.py+ [ ... spider starts crawling and appends the result to items.json ... ]+ .. command:: version versionscrapy/commands/__init__.pyindex d9c919db9..7cf5d989f 100644@@ -17,13 +17,44 @@ from twisted.python import failure from scrapy.exceptions import ScrapyDeprecationWarning, UsageError from scrapy.utils.conf import arglist_to_dict, feed_process_params_from_cli from scrapy.utils.deprecate import method_is_overridden+from scrapy.utils.project import inside_project from scrapy.utils.python import global_object_name if TYPE_CHECKING: from collections.abc import Iterable + from scrapy import Spider from scrapy.crawler import Crawler, CrawlerProcessBase from scrapy.settings import Settings+ from scrapy.spiderloader import SpiderLoaderProtocol+++def _looks_like_spider_file(arg: str) -> bool:+ path = Path(arg)+ if path.suffix in {".py", ".pyw"}:+ return True+ try:+ return path.is_file()+ except (OSError, ValueError): # e.g. a name too long to be a path+ return False+++def _load_spider(spider_loader: SpiderLoaderProtocol, name: str) -> type[Spider]:+ """Return the spider class named *name*, raising :exc:`UsageError` with+ a hint about how to proceed if there is no such spider."""+ try:+ return spider_loader.load(name)+ except KeyError:+ if _looks_like_spider_file(name):+ hint = f"To run a spider defined in a file, use: scrapy runspider {name}"+ elif inside_project():+ hint = 'Use "scrapy list" to see available spiders.'+ else:+ hint = (+ "No active project: spiders can only be referenced by name "+ "from within a project."+ )+ raise UsageError(f"Spider not found: {name}\n{hint}\n") from None class ScrapyCommand(ABC):scrapy/commands/check.pyindex 2113c19d2..01cf4f7af 100644@@ -7,7 +7,7 @@ from unittest import TextTestResult as _TextTestResult from unittest import TextTestRunner from scrapy import Spider-from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _load_spider from scrapy.contracts import ContractsManager from scrapy.utils.conf import build_component_list from scrapy.utils.misc import load_object, set_environ@@ -92,7 +92,7 @@ class Command(ScrapyCommand): with set_environ(SCRAPY_CHECK="true"): for spidername in args or spider_loader.list():- spidercls = spider_loader.load(spidername)+ spidercls = _load_spider(spider_loader, spidername) spidercls.start = start # type: ignore[method-assign] tested_methods = conman.tested_methods_from_spidercls(spidercls)scrapy/commands/crawl.pyindex 866ba9f6b..62470a98e 100644@@ -2,7 +2,7 @@ from __future__ import annotations from typing import TYPE_CHECKING -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import BaseRunSpiderCommand, _load_spider from scrapy.exceptions import UsageError if TYPE_CHECKING:@@ -16,7 +16,17 @@ class Command(BaseRunSpiderCommand): return "[options] <spider>" def short_desc(self) -> str:- return "Run a spider"+ return "Run a spider of the current project"++ def long_desc(self) -> str:+ return (+ "Run a spider of the current project, given its name. "+ 'Use "scrapy list" to see available spiders, and '+ '"scrapy runspider" to run a spider defined in a standalone file. '+ "Scraped items are not stored anywhere unless you use -o/-O "+ "(e.g. -o items.json, or -o -:jsonl for standard output) or "+ "configure the FEEDS setting."+ ) def run(self, args: list[str], opts: argparse.Namespace) -> None: if len(args) < 1:@@ -28,7 +38,8 @@ class Command(BaseRunSpiderCommand): spname = args[0] assert self.crawler_process- self.crawler_process.crawl(spname, **opts.spargs)+ spidercls = _load_spider(self.crawler_process.spider_loader, spname)+ self.crawler_process.crawl(spidercls, **opts.spargs) self.crawler_process.start() if self.crawler_process.bootstrap_failed: self.exitcode = 1scrapy/commands/fetch.pyindex 0b8311efb..35a19cc3c 100644@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, Any from w3lib.url import is_url -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _load_spider from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils.datatypes import SequenceExclude@@ -86,7 +86,7 @@ class Command(ScrapyCommand): assert self.crawler_process spider_loader = self.crawler_process.spider_loader if opts.spider:- spidercls = spider_loader.load(opts.spider)+ spidercls = _load_spider(spider_loader, opts.spider) else: spidercls = spidercls_for_request(spider_loader, request, spidercls) scrapy/commands/parse.pyindex 93194ded7..f74dbd0b5 100644@@ -10,7 +10,7 @@ from itemadapter import ItemAdapter from twisted.internet.defer import Deferred, maybeDeferred from w3lib.url import is_url -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import BaseRunSpiderCommand, _load_spider from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils import display@@ -246,12 +246,7 @@ class Command(BaseRunSpiderCommand): assert self.crawler_process spider_loader = self.crawler_process.spider_loader if opts.spider:- try:- self.spidercls = spider_loader.load(opts.spider)- except KeyError:- logger.error(- "Unable to find spider: %(spider)s", {"spider": opts.spider}- )+ self.spidercls = _load_spider(spider_loader, opts.spider) else: self.spidercls = spidercls_for_request(spider_loader, Request(url)) if not self.spidercls:scrapy/commands/runspider.pyindex 0b9036457..50003cc50 100644@@ -1,13 +1,16 @@ from __future__ import annotations import sys+import warnings from importlib import import_module from pathlib import Path from typing import TYPE_CHECKING, Any, ClassVar from scrapy.commands import BaseRunSpiderCommand from scrapy.exceptions import UsageError-from scrapy.spiderloader import DummySpiderLoader+from scrapy.spiderloader import DummySpiderLoader, SpiderLoader, get_spider_loader+from scrapy.utils.misc import load_object+from scrapy.utils.project import inside_project from scrapy.utils.spider import iter_spider_classes if TYPE_CHECKING:@@ -41,13 +44,44 @@ class Command(BaseRunSpiderCommand): return "Run a self-contained spider (without creating a project)" def long_desc(self) -> str:- return "Run the spider defined in the given file"+ return (+ "Run the spider defined in the given Python file, without needing "+ "a project. To run a spider of the current project by name, use "+ '"scrapy crawl" instead. Scraped items are not stored anywhere '+ "unless you use -o/-O (e.g. -o items.json, or -o -:jsonl for "+ "standard output) or configure the FEEDS setting."+ )++ def _project_has_spider(self, name: str) -> bool:+ """Return whether *name* is the name of a spider of the active+ project, if any."""+ if not inside_project():+ return False+ assert self.settings is not None+ settings = self.settings.copy()+ # Spider modules that fail to import must not hide the original error.+ settings.set("SPIDER_LOADER_WARN_ONLY", True, priority="cmdline")+ if load_object(settings["SPIDER_LOADER_CLASS"]) is DummySpiderLoader:+ settings.set("SPIDER_LOADER_CLASS", SpiderLoader, priority="cmdline")+ try:+ with warnings.catch_warnings():+ warnings.simplefilter("ignore")+ spider_names = get_spider_loader(settings).list()+ except Exception: # only a hint, it must not hide the original error+ return False+ return name in spider_names def run(self, args: list[str], opts: argparse.Namespace) -> None: if len(args) != 1: raise UsageError filename = Path(args[0]) if not filename.exists():+ if self._project_has_spider(args[0]):+ raise UsageError(+ f"File not found: {filename}\n"+ f"{args[0]!r} is a spider of the current project, run it "+ f"with: scrapy crawl {args[0]}\n"+ ) raise UsageError(f"File not found: {filename}\n") try: module = _import_file(filename)scrapy/commands/shell.pyindex 19138ffd0..0bde97d80 100644@@ -10,7 +10,7 @@ import asyncio from threading import Thread from typing import TYPE_CHECKING, Any, ClassVar -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _load_spider from scrapy.crawler import AsyncCrawlerProcess, Crawler from scrapy.http import Request from scrapy.shell import Shell@@ -75,7 +75,7 @@ class Command(ScrapyCommand): spidercls: type[Spider] = DefaultSpider if opts.spider:- spidercls = spider_loader.load(opts.spider)+ spidercls = _load_spider(spider_loader, opts.spider) elif url: spidercls = spidercls_for_request( spider_loader, Request(url), spidercls, log_multiple=Truetests/test_commands.pyindex 3e687e811..006590ceb 100644@@ -665,3 +665,111 @@ class TestPopCommandName: command = _pop_command_name(argv) assert command == "crawl" assert argv == ["scrapy", "-h"]+++class TestSpiderNotFound(TestProjectBase):+ """Commands that take a spider name report unknown spiders as usage+ errors with a hint, instead of a traceback."""++ spider_code = """+import scrapy++class MySpider(scrapy.Spider):+ name = 'myspider'+"""++ def _write_spider(self, proj_path: Path) -> Path:+ path = proj_path / self.project_name / "spiders" / "myspider.py"+ path.write_text(self.spider_code, encoding="utf-8")+ return path++ @pytest.mark.parametrize(+ "args",+ [+ ("crawl", "nonexistent"),+ ("check", "nonexistent"),+ ("fetch", "--spider", "nonexistent", "http://localhost:1/"),+ ("shell", "--spider", "nonexistent", "http://localhost:1/"),+ ("parse", "--spider", "nonexistent", "http://localhost:1/"),+ ],+ )+ def test_unknown_spider_name(+ self, proj_path: Path, args: tuple[str, ...]+ ) -> None:+ self._write_spider(proj_path)+ returncode, _, err = proc(*args, cwd=proj_path)+ assert returncode == 2+ assert "Spider not found: nonexistent" in err+ assert 'Use "scrapy list" to see available spiders.' in err+ assert "Traceback" not in err++ def test_crawl_spider_file(self, proj_path: Path) -> None:+ spider_path = self._write_spider(proj_path)+ rel_path = str(spider_path.relative_to(proj_path))+ returncode, _, err = proc("crawl", rel_path, cwd=proj_path)+ assert returncode == 2+ assert f"Spider not found: {rel_path}" in err+ assert (+ f"To run a spider defined in a file, use: scrapy runspider {rel_path}"+ in err+ )+ assert "Traceback" not in err++ def test_fetch_unknown_spider_outside_project(self, tmp_path: Path) -> None:+ returncode, _, err = proc(+ "fetch", "--spider", "nonexistent", "http://localhost:1/", cwd=tmp_path+ )+ assert returncode == 2+ assert "Spider not found: nonexistent" in err+ assert "No active project" in err+ assert "scrapy list" not in err+ assert "Traceback" not in err++ def test_runspider_project_spider_name(self, proj_path: Path) -> None:+ self._write_spider(proj_path)+ returncode, _, err = proc("runspider", "myspider", cwd=proj_path)+ assert returncode == 2+ assert "File not found: myspider" in err+ assert "run it with: scrapy crawl myspider" in err++ def test_runspider_unknown_name_in_project(self, proj_path: Path) -> None:+ self._write_spider(proj_path)+ returncode, _, err = proc("runspider", "nonexistent", cwd=proj_path)+ assert returncode == 2+ assert "File not found: nonexistent" in err+ assert "scrapy crawl" not in err++ @pytest.mark.parametrize(+ "broken_code",+ [+ "import nonexistent_module_for_test\n",+ "raise RuntimeError('broken spider module')\n",+ ],+ )+ def test_runspider_project_spider_name_broken_module(+ self, proj_path: Path, broken_code: str+ ) -> None:+ self._write_spider(proj_path)+ (proj_path / self.project_name / "spiders" / "broken.py").write_text(+ broken_code, encoding="utf-8"+ )+ returncode, _, err = proc("runspider", "myspider", cwd=proj_path)+ assert returncode == 2+ assert "File not found: myspider" in err+ assert "Traceback" not in err++ def test_crawl_unknown_spider_long_name(self, proj_path: Path) -> None:+ self._write_spider(proj_path)+ name = "a" * 300+ returncode, _, err = proc("crawl", name, cwd=proj_path)+ assert returncode == 2+ assert f"Spider not found: {name}" in err+ assert "Traceback" not in err++ def test_help_distinguishes_crawl_and_runspider(self, proj_path: Path) -> None:+ _, crawl_help, _ = proc("crawl", "-h", cwd=proj_path)+ assert "scrapy runspider" in crawl_help+ assert "-o" in crawl_help+ _, runspider_help, _ = proc("runspider", "-h", cwd=proj_path)+ assert "scrapy crawl" in runspider_help+ assert "-o" in runspider_help
docs/topics/commands.rstindex e8a843e80..343193627 100644@@ -114,8 +114,8 @@ some usage help and the available commands:: scrapy <command> [options] [args] Available commands:- crawl Run a spider fetch Fetch a URL using the Scrapy downloader+ runspider Run a spider from a Python file, no project required [...] The first line will print the currently active project if you're inside a@@ -263,7 +263,9 @@ crawl * Syntax: ``scrapy crawl <spider>`` * Requires project: *yes* -Start crawling using a spider.+Start crawling using the spider with the given :attr:`~scrapy.Spider.name`,+which must be one of those that :command:`list` reports. To run a spider from a+file instead, use :command:`runspider`. Supported options: @@ -571,8 +573,9 @@ runspider * Syntax: ``scrapy runspider <spider_file.py>`` * Requires project: *no* -Run a spider self-contained in a Python file, without having to create a-project.+Run the spider defined in the given Python file, without requiring a project.++Supported options: the same as :command:`crawl`. Example usage:: scrapy/commands/crawl.pyindex 866ba9f6b..4e086e057 100644@@ -16,7 +16,7 @@ class Command(BaseRunSpiderCommand): return "[options] <spider>" def short_desc(self) -> str:- return "Run a spider"+ return "Run a spider of the current project, by name" def run(self, args: list[str], opts: argparse.Namespace) -> None: if len(args) < 1:scrapy/commands/runspider.pyindex 0b9036457..9cdb393ab 100644@@ -38,7 +38,7 @@ class Command(BaseRunSpiderCommand): return "[options] <spider_file>" def short_desc(self) -> str:- return "Run a self-contained spider (without creating a project)"+ return "Run a spider from a Python file, no project required" def long_desc(self) -> str: return "Run the spider defined in the given file"