Backtestsscrapyrun-20261025-211112#36
Add command line option for doing POSTs
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches add an option to fetch, view (via fetch), shell and parse, but Patch A delivers what the issue asks for: a `--data` option that behaves like `curl -d` (POST by default, body set, `Content-Type: application/x-www-form-urlencoded`), plus `-X/--method` to override the method. The independent tests confirm it: A passes all five scenarios, while B fails the form-encoding check because `Request.from_curl` does not add the Content-Type header that real `curl -d` sends, so servers expecting form data may not parse B's POST body. B's `--curl COMMAND` design is a reasonable, more general alternative, but it forces users to embed a whole curl command string rather than offering the simple data flag the issue requests, and its parse-command test does not exercise a POST at all. A's tests cover each command (fetch, parse, shell, view, including the reactorless shell path and the method/data interaction and empty-method validation); the view test is skipped only on Windows, which is a reasonable constraint for a fake-browser script and still runs on the platforms the issue implies. On craft, A adds a news entry and consistent docs for every command; B imports `SequenceExclude` from `scrapy.utils.datatypes` while the neighbouring fetch command uses `scrapy.utils._datatypes`, suggesting a deprecated import path, and changes the `set_spidercls` signature of the parse command. B's usage-error handling and URL/curl exclusivity checks are fine but do not outweigh the functional gap. A is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
25m 19s
Wall-clock time in the container
Full test suite
No new failures
249 passed · 5 skipped · 254 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
254vs 215 merged
Changed lines, ours vs merged
The issue
View on GitHubPreviously reported by pablo on Trac http://dev.scrapy.org/ticket/292
It would be nice to add a command line option for performing POSTs, like curl -d does.
This would be added to all commands that accepts a url argument, like crawl, parse, shell, fetch and view.
The two patches
docs/news.rstindex c133a5f53..b5f77ee31 100644@@ -69,6 +69,14 @@ Backward-incompatible changes effect. (:gh:`8181`) +New features+~~~~~~~~~~~~++- The :command:`fetch`, :command:`view`, :command:`shell` and+ :command:`parse` commands now support ``-X``/``--method`` and ``--data``+ options to set the HTTP method and body of the request sent for the given+ URL, e.g. to send POST requests like ``curl -d`` does.+ .. _release-2.19.0: Scrapy 2.19.0 (2026-09-10)docs/topics/commands.rstindex b9b7c2a31..962ba6d03 100644@@ -462,11 +462,21 @@ Supported options: * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+ ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+ implies ``POST`` (unless ``--method`` is given) and sets the+ ``Content-Type: application/x-www-form-urlencoded`` request header+ Usage examples:: $ scrapy fetch --nolog http://www.example.com/some/page.html [ ... html content here ... ] + $ scrapy fetch --nolog --data "foo=bar" http://www.example.com/some/form+ [ ... response to the POST request ... ]+ $ scrapy fetch --nolog --headers http://www.example.com/ > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > Accept-Language: en@@ -500,6 +510,13 @@ Supported options: * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+ ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+ implies ``POST`` (unless ``--method`` is given) and sets the+ ``Content-Type: application/x-www-form-urlencoded`` request header+ Usage example:: $ scrapy view http://www.example.com/some/page.html@@ -528,6 +545,15 @@ Supported options: this only affects the URL you may pass as argument on the command line; once you are inside the shell, ``fetch(url)`` will still follow HTTP redirects by default. +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+ ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+ implies ``POST`` (unless ``--method`` is given) and sets the+ ``Content-Type: application/x-www-form-urlencoded`` request header;+ like ``--no-redirect``, ``-X`` and ``--data`` only affect the URL passed as+ argument on the command line, and require it+ Usage example:: $ scrapy shell http://www.example.com/some/page.html@@ -593,6 +619,13 @@ Supported options: * ``--output`` or ``-o``: dump scraped items to a file +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+ ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+ implies ``POST`` (unless ``--method`` is given) and sets the+ ``Content-Type: application/x-www-form-urlencoded`` request header+ .. skip: start Usage example::scrapy/commands/__init__.pyindex c87c390c1..d9ce4617e 100644@@ -179,6 +179,42 @@ class ScrapyCommand(ABC): raise NotImplementedError +def _add_request_options(parser: argparse.ArgumentParser) -> None:+ """Add options to set the method and body of the request built from the+ URL passed to a command."""+ parser.add_argument(+ "-X",+ "--method",+ dest="method",+ metavar="METHOD",+ help="HTTP method of the request (default: POST if --data is given, "+ "GET otherwise)",+ )+ parser.add_argument(+ "--data",+ dest="data",+ metavar="DATA",+ help="send DATA as the request body, like curl -d; implies POST "+ "and an application/x-www-form-urlencoded Content-Type header",+ )+++def _request_kwargs_from_opts(opts: argparse.Namespace) -> dict[str, Any]:+ """Return the Request keyword arguments for the options added by+ :func:`_add_request_options`."""+ kwargs: dict[str, Any] = {}+ if opts.method is not None:+ method = opts.method.strip().upper()+ if not method:+ raise UsageError("Invalid -X/--method value: empty", print_help=False)+ kwargs["method"] = method+ if opts.data is not None:+ kwargs.setdefault("method", "POST")+ kwargs["body"] = opts.data+ kwargs["headers"] = {"Content-Type": "application/x-www-form-urlencoded"}+ return kwargs++ class BaseRunSpiderCommand(ScrapyCommand, ABC): """ Common class used to share functionality between the crawl, parse and runspider commandsscrapy/commands/fetch.pyindex c345e297c..ce59e17f8 100644@@ -6,7 +6,11 @@ from typing import TYPE_CHECKING, Any from w3lib.url import is_url -from scrapy.commands import ScrapyCommand+from scrapy.commands import (+ ScrapyCommand,+ _add_request_options,+ _request_kwargs_from_opts,+) from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils._datatypes import SequenceExclude@@ -48,6 +52,7 @@ class Command(ScrapyCommand): default=False, help="do not handle HTTP 3xx status codes and print response as-is", )+ _add_request_options(parser) def _print_headers(self, headers: dict[bytes, list[bytes]], prefix: bytes) -> None: for key, values in headers.items():@@ -74,6 +79,7 @@ class Command(ScrapyCommand): callback=self._print_response, cb_kwargs={"opts": opts}, dont_filter=True,+ **_request_kwargs_from_opts(opts), ) # by default, let the framework handle redirects, # i.e. command handles all codes expect 3xxscrapy/commands/parse.pyindex 4edadfa58..9d0cf7f3f 100644@@ -10,7 +10,11 @@ from itemadapter import ItemAdapter from twisted.internet.defer import Deferred, maybeDeferred from w3lib.url import is_url -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import (+ BaseRunSpiderCommand,+ _add_request_options,+ _request_kwargs_from_opts,+) from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils import _colorize@@ -119,6 +123,7 @@ class Command(BaseRunSpiderCommand): action="store_true", help="print each depth level one by one", )+ _add_request_options(parser) @property def max_level(self) -> int:@@ -244,6 +249,7 @@ class Command(BaseRunSpiderCommand): def set_spidercls(self, url: str, opts: argparse.Namespace) -> None: assert self.crawler_process+ request_kwargs = _request_kwargs_from_opts(opts) spider_loader = self.crawler_process.spider_loader if opts.spider: try:@@ -258,7 +264,7 @@ class Command(BaseRunSpiderCommand): logger.error("Unable to find spider for: %(url)s", {"url": url}) async def start(spider: Spider) -> AsyncIterator[Any]:- yield self.prepare_request(spider, Request(url), opts)+ yield self.prepare_request(spider, Request(url, **request_kwargs), opts) if self.spidercls: self.spidercls.start = start # type: ignore[assignment,method-assign]scrapy/commands/shell.pyindex f64f557b5..72a547ea8 100644@@ -10,9 +10,14 @@ import asyncio from threading import Thread from typing import TYPE_CHECKING, Any, ClassVar -from scrapy.commands import ScrapyCommand+from scrapy.commands import (+ ScrapyCommand,+ _add_request_options,+ _request_kwargs_from_opts,+) from scrapy.core.engine import ExecutionEngine from scrapy.crawler import AsyncCrawlerProcess, Crawler+from scrapy.exceptions import UsageError from scrapy.http import Request from scrapy.shell import Shell from scrapy.utils.spider import DefaultSpider, spidercls_for_request@@ -59,6 +64,7 @@ class Command(ScrapyCommand): default=False, help="do not handle HTTP 3xx status codes and print response as-is", )+ _add_request_options(parser) def update_vars(self, vars: dict[str, Any]) -> None: # noqa: A002 """You can use this function to update the Scrapy objects that will be@@ -67,6 +73,9 @@ class Command(ScrapyCommand): def run(self, args: list[str], opts: Namespace) -> None: url = args[0] if args else None+ request_kwargs = _request_kwargs_from_opts(opts)+ if request_kwargs and not url:+ raise UsageError("-X/--method and --data require a URL", print_help=False) if url: # first argument may be a local file url = guess_scheme(url)@@ -93,7 +102,7 @@ class Command(ScrapyCommand): self._init_without_reactor(crawler) loop = self._get_reactorless_loop() shell = Shell(crawler, update_vars=self.update_vars, code=opts.code, loop=loop)- shell.start(url=url, redirect=not opts.no_redirect)+ shell.start(url=url, redirect=not opts.no_redirect, **request_kwargs) def _init_with_reactor(self, crawler: Crawler) -> None: # Create the engine in the main threadscrapy/shell.pyindex 019badb59..f85e7e1d1 100644@@ -137,11 +137,12 @@ class Shell: response: Response | None = None, spider: Spider | None = None, redirect: bool = True,+ **kwargs: Any, ) -> None: # disable accidental Ctrl-C key press from shutting down the engine signal.signal(signal.SIGINT, signal.SIG_IGN) if url:- self.fetch(url, spider, redirect=redirect)+ self.fetch(url, spider, redirect=redirect, **kwargs) elif request: self.fetch(request, spider) elif response:tests/test_command_fetch.pyindex e25540f50..d8d0df1b4 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json from typing import TYPE_CHECKING import pytest@@ -42,6 +43,39 @@ class TestFetchCommand: assert "Server: TwistedWeb" in out assert "Content-Type: text/plain" in out + def test_data(self, mockserver: MockServer) -> None:+ returncode, out, err = proc(+ "fetch", "--data", "a=1&b=2", mockserver.url("/echo")+ )+ assert returncode == 0, err+ echo = json.loads(out)+ assert echo["body"] == "a=1&b=2"+ assert echo["headers"]["Content-Type"] == ["application/x-www-form-urlencoded"]++ def test_data_headers(self, mockserver: MockServer) -> None:+ _, out, _ = proc("fetch", "--headers", "--data", "a=1", mockserver.url("/echo"))+ out = out.replace("\r", "") # required on win32+ assert "> Content-Type: application/x-www-form-urlencoded" in out++ @pytest.mark.parametrize("option", ["-X", "--method"])+ def test_method(self, option: str, mockserver: MockServer) -> None:+ # /echo only supports GET and POST, so a PUT gets a 501 response+ _, _, err = proc("fetch", option, "put", mockserver.url("/echo"))+ assert "downloader/request_method_count/PUT" in err+ assert "downloader/response_status_count/501" in err++ def test_method_get_with_data(self, mockserver: MockServer) -> None:+ _, out, err = proc(+ "fetch", "-X", "GET", "--data", "a=1", mockserver.url("/echo")+ )+ assert "downloader/request_method_count/GET" in err+ assert json.loads(out)["body"] == "a=1"++ def test_empty_method(self, mockserver: MockServer) -> None:+ returncode, _, err = proc("fetch", "-X", " ", mockserver.url("/echo"))+ assert returncode == 2+ assert "Invalid -X/--method value" in err+ def test_no_reactor(self, mockserver: MockServer) -> None: _, out, _ = proc( "fetch", "-s", "TWISTED_REACTOR_ENABLED=False", mockserver.url("/text")tests/test_command_parse.pyindex f06816b8c..23255c47b 100644@@ -119,6 +119,9 @@ class MySpider(scrapy.Spider): else: self.logger.debug('It Does Not Work :(') + def parse_request_method(self, response):+ self.logger.debug(f'Request: {{response.request.method}} {{response.json()["body"]}}')+ def parse_request_without_meta(self, response): foo = response.meta.get('foo', 'bar') @@ -306,6 +309,52 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}} ) assert "DEBUG: It Works!" in stderr + def test_request_with_data(self, proj_path: Path, mockserver: MockServer) -> None:+ _, _, stderr = proc(+ "parse",+ "--spider",+ self.spider_name,+ "--data",+ "a=1&b=2",+ "-c",+ "parse_request_method",+ mockserver.url("/echo"),+ cwd=proj_path,+ )+ assert "DEBUG: Request: POST a=1&b=2" in stderr++ @pytest.mark.parametrize("option", ["-X", "--method"])+ def test_request_with_method(+ self, option: str, proj_path: Path, mockserver: MockServer+ ) -> None:+ _, _, stderr = proc(+ "parse",+ "--spider",+ self.spider_name,+ option,+ "post",+ "-c",+ "parse_request_method",+ mockserver.url("/echo"),+ cwd=proj_path,+ )+ assert "DEBUG: Request: POST " in stderr++ def test_request_with_empty_method(+ self, proj_path: Path, mockserver: MockServer+ ) -> None:+ returncode, _, stderr = proc(+ "parse",+ "--spider",+ self.spider_name,+ "-X",+ "",+ mockserver.url("/echo"),+ cwd=proj_path,+ )+ assert returncode == 2+ assert "Invalid -X/--method value" in stderr+ def test_pipelines(self, proj_path: Path, mockserver: MockServer) -> None: _, _, stderr = proc( "parse",tests/test_command_shell.pyindex a483f5abd..9e4f81b42 100644@@ -102,6 +102,45 @@ class TestShellCommand: assert ret == 0, out assert "Crawled (302)" in err + def test_data(self, mockserver: MockServer) -> None:+ code = "(request.method, request.body, response.json()['body'])"+ _, out, _ = proc(+ "shell", "--data", "a=1&b=2", mockserver.url("/echo"), "-c", code+ )+ assert out.strip() == "('POST', b'a=1&b=2', 'a=1&b=2')"++ def test_data_no_reactor(self, mockserver: MockServer) -> None:+ code = "(request.method, response.json()['body'])"+ _, out, _ = proc(+ "shell",+ "-s",+ "TWISTED_REACTOR_ENABLED=False",+ "--data",+ "a=1",+ mockserver.url("/echo"),+ "-c",+ code,+ )+ assert out.strip() == "('POST', 'a=1')"++ @pytest.mark.parametrize("option", ["-X", "--method"])+ def test_method(self, option: str, mockserver: MockServer) -> None:+ _, out, _ = proc(+ "shell",+ option,+ "put",+ mockserver.url("/echo"),+ "-c",+ "(request.method, response.status)",+ )+ assert out.strip() == "('PUT', 501)"++ @pytest.mark.parametrize("args", [("--data", "a=1"), ("-X", "POST")])+ def test_request_options_without_url(self, args: tuple[str, ...]) -> None:+ returncode, _, err = proc("shell", *args, "-c", "item")+ assert returncode == 2+ assert "-X/--method and --data require a URL" in err+ def test_request_replace(self, mockserver: MockServer) -> None: url = mockserver.url("/text") code = f"fetch('{url}') or fetch(response.request.replace(method='POST'))"tests/test_commands.pyindex db07a3d4d..088d6d699 100644@@ -883,6 +883,27 @@ class TestViewCommand: body = Path(url.removeprefix("file://")).read_text(encoding="utf-8") assert "<p class='one'>Works</p>" in body + @pytest.mark.skipif(+ sys.platform == "win32", reason="requires a POSIX shell browser script"+ )+ def test_view_data(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, mockserver: MockServer+ ) -> None:+ opened = tmp_path / "opened.txt"+ browser = tmp_path / "fake-browser.sh"+ write_recording_browser(browser, opened)+ monkeypatch.setenv("BROWSER", str(browser))++ returncode, _, err = proc(+ "view", "--data", "a=1&b=2", mockserver.url("/echo"), cwd=tmp_path+ )++ assert returncode == 0, err+ assert "downloader/request_method_count/POST" in err+ url = opened.read_text(encoding="utf-8")+ body = Path(url.removeprefix("file://")).read_text(encoding="utf-8")+ assert '"body": "a=1&b=2"' in body+ def test_non_text_response(self, mockserver: MockServer) -> None: returncode, _, err = proc( "view", mockserver.url("/static/files/images/scrapy.png")
docs/topics/commands.rstindex b9b7c2a31..1d56ee0aa 100644@@ -462,11 +462,21 @@ Supported options: * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+ URL argument, taking the URL, HTTP method, headers, cookies and body from the+ command (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It cannot+ be combined with a URL argument.++ .. versionadded:: VERSION+ Usage examples:: $ scrapy fetch --nolog http://www.example.com/some/page.html [ ... html content here ... ] + $ scrapy fetch --nolog --curl 'curl -d title=hello https://httpbin.org/post'+ [ ... html content here ... ]+ $ scrapy fetch --nolog --headers http://www.example.com/ > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > Accept-Language: en@@ -500,6 +510,12 @@ Supported options: * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+ URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It+ cannot be combined with a URL argument.++ .. versionadded:: VERSION+ Usage example:: $ scrapy view http://www.example.com/some/page.html@@ -528,6 +544,12 @@ Supported options: this only affects the URL you may pass as argument on the command line; once you are inside the shell, ``fetch(url)`` will still follow HTTP redirects by default. +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+ URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It+ cannot be combined with a URL argument.++ .. versionadded:: VERSION+ Usage example:: $ scrapy shell http://www.example.com/some/page.html@@ -566,6 +588,12 @@ Supported options: * ``--callback`` or ``-c``: spider method to use as callback for parsing the response +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+ URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It+ cannot be combined with a URL argument.++ .. versionadded:: VERSION+ * ``--meta`` or ``-m``: additional request meta that will be passed to the callback request. This must be a valid JSON string. Example: ``--meta='{"foo": "bar"}'``@@ -877,3 +905,5 @@ Example: .. code-block:: python COMMANDS_MODULE = "mybot.commands"++.. _cURL: https://curl.se/scrapy/commands/__init__.pyindex c87c390c1..7805d5e2c 100644@@ -17,6 +17,7 @@ from typing import TYPE_CHECKING, Any, ClassVar from scrapy import signals from scrapy.exceptions import ScrapyDeprecationWarning, UsageError from scrapy.extensions.feedexport import FeedExporter+from scrapy.http import Request from scrapy.utils.conf import arglist_to_dict, feed_process_params_from_cli from scrapy.utils.deprecate import method_is_overridden from scrapy.utils.python import global_object_name@@ -32,6 +33,27 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +def _add_curl_option(parser: argparse.ArgumentParser) -> None:+ """Register the ``--curl`` option on commands that accept a URL."""+ parser.add_argument(+ "--curl",+ metavar="COMMAND",+ default=None,+ help="build the request from a curl command, e.g. "+ "--curl 'curl -d a=1 https://example.com'; cannot be combined "+ "with a URL argument",+ )+++def _request_from_curl(curl_command: str, **kwargs: Any) -> Request:+ """Build a :class:`~scrapy.Request` from a curl command, reporting parsing+ errors as :exc:`~scrapy.exceptions.UsageError`."""+ try:+ return Request.from_curl(curl_command, **kwargs)+ except ValueError as e:+ raise UsageError(str(e), print_help=False) from e++ class _PdbHandler(logging.Handler): """Start a post-mortem debugging session on every logged record that comes with an exception traceback."""scrapy/commands/fetch.pyindex c345e297c..a5b44c562 100644@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, Any from w3lib.url import is_url -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _add_curl_option, _request_from_curl from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils._datatypes import SequenceExclude@@ -34,6 +34,7 @@ class Command(ScrapyCommand): def add_options(self, parser: ArgumentParser) -> None: super().add_options(parser)+ _add_curl_option(parser) parser.add_argument("--spider", dest="spider", help="use this spider") parser.add_argument( "--headers",@@ -67,14 +68,24 @@ class Command(ScrapyCommand): sys.stdout.buffer.write(bytes_ + b"\n") def run(self, args: list[str], opts: Namespace) -> None:- if len(args) != 1 or not is_url(args[0]):- raise UsageError- request = Request(- args[0],- callback=self._print_response,- cb_kwargs={"opts": opts},- dont_filter=True,- )+ if opts.curl:+ if args:+ raise UsageError("--curl cannot be combined with a URL argument")+ request = _request_from_curl(+ opts.curl,+ callback=self._print_response,+ cb_kwargs={"opts": opts},+ dont_filter=True,+ )+ else:+ if len(args) != 1 or not is_url(args[0]):+ raise UsageError+ request = Request(+ args[0],+ callback=self._print_response,+ cb_kwargs={"opts": opts},+ dont_filter=True,+ ) # by default, let the framework handle redirects, # i.e. command handles all codes expect 3xx if not opts.no_redirect:scrapy/commands/parse.pyindex 4edadfa58..be84af063 100644@@ -10,7 +10,7 @@ from itemadapter import ItemAdapter from twisted.internet.defer import Deferred, maybeDeferred from w3lib.url import is_url -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import BaseRunSpiderCommand, _add_curl_option, _request_from_curl from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils import _colorize@@ -53,6 +53,7 @@ class Command(BaseRunSpiderCommand): def add_options(self, parser: argparse.ArgumentParser) -> None: super().add_options(parser)+ _add_curl_option(parser) parser.add_argument( "--spider", dest="spider",@@ -242,7 +243,7 @@ class Command(BaseRunSpiderCommand): ) return None - def set_spidercls(self, url: str, opts: argparse.Namespace) -> None:+ def set_spidercls(self, request: Request, opts: argparse.Namespace) -> None: assert self.crawler_process spider_loader = self.crawler_process.spider_loader if opts.spider:@@ -253,12 +254,12 @@ class Command(BaseRunSpiderCommand): "Unable to find spider: %(spider)s", {"spider": opts.spider} ) else:- self.spidercls = spidercls_for_request(spider_loader, Request(url))+ self.spidercls = spidercls_for_request(spider_loader, request) if not self.spidercls:- logger.error("Unable to find spider for: %(url)s", {"url": url})+ logger.error("Unable to find spider for: %(url)s", {"url": request.url}) async def start(spider: Spider) -> AsyncIterator[Any]:- yield self.prepare_request(spider, Request(url), opts)+ yield self.prepare_request(spider, request, opts) if self.spidercls: self.spidercls.start = start # type: ignore[assignment,method-assign]@@ -404,12 +405,18 @@ class Command(BaseRunSpiderCommand): def run(self, args: list[str], opts: argparse.Namespace) -> None: # parse arguments- if not len(args) == 1 or not is_url(args[0]):- raise UsageError- url = args[0]+ if opts.curl:+ if args:+ raise UsageError("--curl cannot be combined with a URL argument")+ request = _request_from_curl(opts.curl)+ else:+ if not len(args) == 1 or not is_url(args[0]):+ raise UsageError+ request = Request(args[0])+ url = request.url # prepare spidercls- self.set_spidercls(url, opts)+ self.set_spidercls(request, opts) if self.spidercls and opts.depth > 0: self.start_parsing(url, opts)scrapy/commands/shell.pyindex f64f557b5..4caa43719 100644@@ -10,11 +10,13 @@ import asyncio from threading import Thread from typing import TYPE_CHECKING, Any, ClassVar -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _add_curl_option, _request_from_curl from scrapy.core.engine import ExecutionEngine from scrapy.crawler import AsyncCrawlerProcess, Crawler+from scrapy.exceptions import UsageError from scrapy.http import Request from scrapy.shell import Shell+from scrapy.utils.datatypes import SequenceExclude from scrapy.utils.spider import DefaultSpider, spidercls_for_request from scrapy.utils.url import guess_scheme @@ -46,6 +48,7 @@ class Command(ScrapyCommand): def add_options(self, parser: ArgumentParser) -> None: super().add_options(parser)+ _add_curl_option(parser) parser.add_argument( "-c", dest="code",@@ -67,7 +70,19 @@ class Command(ScrapyCommand): def run(self, args: list[str], opts: Namespace) -> None: url = args[0] if args else None- if url:+ request: Request | None = None+ if opts.curl:+ if url:+ raise UsageError("--curl cannot be combined with a URL argument")+ request = _request_from_curl(opts.curl, dont_filter=True)+ if opts.no_redirect:+ request.meta["handle_httpstatus_all"] = True+ else:+ request.meta["handle_httpstatus_list"] = SequenceExclude(+ range(300, 400)+ )+ url = request.url+ elif url: # first argument may be a local file url = guess_scheme(url) @@ -77,6 +92,10 @@ class Command(ScrapyCommand): spidercls: type[Spider] = DefaultSpider if opts.spider: spidercls = spider_loader.load(opts.spider)+ elif request is not None:+ spidercls = spidercls_for_request(+ spider_loader, request, spidercls, log_multiple=True+ ) elif url: spidercls = spidercls_for_request( spider_loader, Request(url), spidercls, log_multiple=True@@ -93,7 +112,10 @@ class Command(ScrapyCommand): self._init_without_reactor(crawler) loop = self._get_reactorless_loop() shell = Shell(crawler, update_vars=self.update_vars, code=opts.code, loop=loop)- shell.start(url=url, redirect=not opts.no_redirect)+ if request is not None:+ shell.start(request=request, redirect=not opts.no_redirect)+ else:+ shell.start(url=url, redirect=not opts.no_redirect) def _init_with_reactor(self, crawler: Crawler) -> None: # Create the engine in the main threadtests/test_command_fetch.pyindex e25540f50..bf239b34b 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json from typing import TYPE_CHECKING import pytest@@ -48,6 +49,23 @@ class TestFetchCommand: ) assert out.strip() == "Works" + def test_curl(self, mockserver: MockServer) -> None:+ url = mockserver.url("/echo")+ _, out, _ = proc("fetch", "--curl", f"curl -d a=1 -H 'X-Test: foo' {url}")+ echo = json.loads(out)+ assert echo["body"] == "a=1"+ assert echo["headers"]["X-Test"] == ["foo"]++ def test_curl_with_url(self, mockserver: MockServer) -> None:+ url = mockserver.url("/echo")+ code, _, _ = proc("fetch", "--curl", f"curl {url}", url)+ assert code != 0++ def test_curl_invalid(self) -> None:+ code, _, err = proc("fetch", "--curl", "not-a-curl-command")+ assert code != 0+ assert "curl" in err+ class TestFetchCommandWithSpider(TestProjectBase): @pytest.fixture(autouse=True)tests/test_command_parse.pyindex f06816b8c..663ea5799 100644@@ -239,6 +239,41 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}} ) assert "DEBUG: It Works!" in stderr + def test_invalid_url(self, proj_path: Path) -> None:+ code, _, _ = proc(+ "parse", "--spider", self.spider_name, "not-a-url", cwd=proj_path+ )+ assert code != 0++ def test_curl(self, proj_path: Path, mockserver: MockServer) -> None:+ _, _, stderr = proc(+ "parse",+ "--spider",+ self.spider_name,+ "-a",+ "test_arg=1",+ "--curl",+ f"curl {mockserver.url('/html')}",+ "-c",+ "parse",+ "--verbose",+ cwd=proj_path,+ )+ assert "DEBUG: It Works!" in stderr++ def test_curl_with_url(self, proj_path: Path, mockserver: MockServer) -> None:+ url = mockserver.url("/html")+ code, _, _ = proc(+ "parse",+ "--spider",+ self.spider_name,+ "--curl",+ f"curl {url}",+ url,+ cwd=proj_path,+ )+ assert code != 0+ def test_request_with_meta(self, proj_path: Path, mockserver: MockServer) -> None: raw_json_string = '{"foo" : "baz"}' _, _, stderr = proc(tests/test_command_shell.pyindex a483f5abd..dc156947a 100644@@ -43,6 +43,34 @@ class TestShellCommand: _, out, _ = proc("shell", mockserver.url("/text"), "-c", "response.body") assert "Works" in out + def test_curl(self, mockserver: MockServer) -> None:+ url = mockserver.url("/echo")+ _, out, _ = proc(+ "shell",+ "--curl",+ f"curl -d a=1 {url}",+ "-c",+ "(request.method, request.body)",+ )+ assert "('POST', b'a=1')" in out++ def test_curl_no_redirect(self, mockserver: MockServer) -> None:+ url = mockserver.url("/redirect-no-meta-refresh")+ _, out, _ = proc(+ "shell",+ "--no-redirect",+ "--curl",+ f"curl {url}",+ "-c",+ "response.status",+ )+ assert out.strip().endswith("302")++ def test_curl_with_url(self, mockserver: MockServer) -> None:+ url = mockserver.url("/echo")+ code, _, _ = proc("shell", "--curl", f"curl {url}", url, "-c", "url")+ assert code != 0+ def test_response_type_text(self, mockserver: MockServer) -> None: _, out, _ = proc("shell", mockserver.url("/text"), "-c", "type(response)") assert "TextResponse" in out