Backtestsscrapyrun-20261025-211112#36

Issue #36 Merged fix PR #7703 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Add command line option for doing POSTs

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches add an option to fetch, view (via fetch), shell and parse, but Patch A delivers what the issue asks for: a `--data` option that behaves like `curl -d` (POST by default, body set, `Content-Type: application/x-www-form-urlencoded`), plus `-X/--method` to override the method. The independent tests confirm it: A passes all five scenarios, while B fails the form-encoding check because `Request.from_curl` does not add the Content-Type header that real `curl -d` sends, so servers expecting form data may not parse B's POST body. B's `--curl COMMAND` design is a reasonable, more general alternative, but it forces users to embed a whole curl command string rather than offering the simple data flag the issue requests, and its parse-command test does not exercise a POST at all. A's tests cover each command (fetch, parse, shell, view, including the reactorless shell path and the method/data interaction and empty-method validation); the view test is skipped only on Windows, which is a reasonable constraint for a fake-browser script and still runs on the platforms the issue implies. On craft, A adds a news entry and consistent docs for every command; B imports `SequenceExclude` from `scrapy.utils.datatypes` while the neighbouring fetch command uses `scrapy.utils._datatypes`, suggesting a deprecated import path, and changes the `set_spidercls` signature of the parse command. B's usage-error handling and URL/curl exclusivity checks are fine but do not outweigh the functional gap. A is the better fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

25m 19s

Wall-clock time in the container

Full test suite

No new failures

249 passed · 5 skipped · 254 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

254vs 215 merged

Changed lines, ours vs merged

Previously reported by pablo on Trac http://dev.scrapy.org/ticket/292

It would be nice to add a command line option for performing POSTs, like curl -d does.

This would be added to all commands that accepts a url argument, like crawl, parse, shell, fetch and view.

The two patches

Our patch+248 −6
docs/news.rstindex c133a5f53..b5f77ee31 100644@@ -69,6 +69,14 @@ Backward-incompatible changes     effect.     (:gh:`8181`) +New features+~~~~~~~~~~~~++-   The :command:`fetch`, :command:`view`, :command:`shell` and+    :command:`parse` commands now support ``-X``/``--method`` and ``--data``+    options to set the HTTP method and body of the request sent for the given+    URL, e.g. to send POST requests like ``curl -d`` does.+ .. _release-2.19.0:  Scrapy 2.19.0 (2026-09-10)docs/topics/commands.rstindex b9b7c2a31..962ba6d03 100644@@ -462,11 +462,21 @@ Supported options:  * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+  ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+  implies ``POST`` (unless ``--method`` is given) and sets the+  ``Content-Type: application/x-www-form-urlencoded`` request header+ Usage examples::      $ scrapy fetch --nolog http://www.example.com/some/page.html     [ ... html content here ... ] +    $ scrapy fetch --nolog --data "foo=bar" http://www.example.com/some/form+    [ ... response to the POST request ... ]+     $ scrapy fetch --nolog --headers http://www.example.com/     > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8     > Accept-Language: en@@ -500,6 +510,13 @@ Supported options:  * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+  ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+  implies ``POST`` (unless ``--method`` is given) and sets the+  ``Content-Type: application/x-www-form-urlencoded`` request header+ Usage example::      $ scrapy view http://www.example.com/some/page.html@@ -528,6 +545,15 @@ Supported options:   this only affects the URL you may pass as argument on the command line;   once you are inside the shell, ``fetch(url)`` will still follow HTTP redirects by default. +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+  ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+  implies ``POST`` (unless ``--method`` is given) and sets the+  ``Content-Type: application/x-www-form-urlencoded`` request header;+  like ``--no-redirect``, ``-X`` and ``--data`` only affect the URL passed as+  argument on the command line, and require it+ Usage example::      $ scrapy shell http://www.example.com/some/page.html@@ -593,6 +619,13 @@ Supported options:  * ``--output`` or ``-o``: dump scraped items to a file +* ``-X METHOD`` or ``--method=METHOD``: HTTP method of the request (default:+  ``POST`` if ``--data`` is given, ``GET`` otherwise)++* ``--data=DATA``: send ``DATA`` as the request body, like ``curl -d``; it+  implies ``POST`` (unless ``--method`` is given) and sets the+  ``Content-Type: application/x-www-form-urlencoded`` request header+ .. skip: start  Usage example::scrapy/commands/__init__.pyindex c87c390c1..d9ce4617e 100644@@ -179,6 +179,42 @@ class ScrapyCommand(ABC):         raise NotImplementedError  +def _add_request_options(parser: argparse.ArgumentParser) -> None:+    """Add options to set the method and body of the request built from the+    URL passed to a command."""+    parser.add_argument(+        "-X",+        "--method",+        dest="method",+        metavar="METHOD",+        help="HTTP method of the request (default: POST if --data is given, "+        "GET otherwise)",+    )+    parser.add_argument(+        "--data",+        dest="data",+        metavar="DATA",+        help="send DATA as the request body, like curl -d; implies POST "+        "and an application/x-www-form-urlencoded Content-Type header",+    )+++def _request_kwargs_from_opts(opts: argparse.Namespace) -> dict[str, Any]:+    """Return the Request keyword arguments for the options added by+    :func:`_add_request_options`."""+    kwargs: dict[str, Any] = {}+    if opts.method is not None:+        method = opts.method.strip().upper()+        if not method:+            raise UsageError("Invalid -X/--method value: empty", print_help=False)+        kwargs["method"] = method+    if opts.data is not None:+        kwargs.setdefault("method", "POST")+        kwargs["body"] = opts.data+        kwargs["headers"] = {"Content-Type": "application/x-www-form-urlencoded"}+    return kwargs++ class BaseRunSpiderCommand(ScrapyCommand, ABC):     """     Common class used to share functionality between the crawl, parse and runspider commandsscrapy/commands/fetch.pyindex c345e297c..ce59e17f8 100644@@ -6,7 +6,11 @@ from typing import TYPE_CHECKING, Any  from w3lib.url import is_url -from scrapy.commands import ScrapyCommand+from scrapy.commands import (+    ScrapyCommand,+    _add_request_options,+    _request_kwargs_from_opts,+) from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils._datatypes import SequenceExclude@@ -48,6 +52,7 @@ class Command(ScrapyCommand):             default=False,             help="do not handle HTTP 3xx status codes and print response as-is",         )+        _add_request_options(parser)      def _print_headers(self, headers: dict[bytes, list[bytes]], prefix: bytes) -> None:         for key, values in headers.items():@@ -74,6 +79,7 @@ class Command(ScrapyCommand):             callback=self._print_response,             cb_kwargs={"opts": opts},             dont_filter=True,+            **_request_kwargs_from_opts(opts),         )         # by default, let the framework handle redirects,         # i.e. command handles all codes expect 3xxscrapy/commands/parse.pyindex 4edadfa58..9d0cf7f3f 100644@@ -10,7 +10,11 @@ from itemadapter import ItemAdapter from twisted.internet.defer import Deferred, maybeDeferred from w3lib.url import is_url -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import (+    BaseRunSpiderCommand,+    _add_request_options,+    _request_kwargs_from_opts,+) from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils import _colorize@@ -119,6 +123,7 @@ class Command(BaseRunSpiderCommand):             action="store_true",             help="print each depth level one by one",         )+        _add_request_options(parser)      @property     def max_level(self) -> int:@@ -244,6 +249,7 @@ class Command(BaseRunSpiderCommand):      def set_spidercls(self, url: str, opts: argparse.Namespace) -> None:         assert self.crawler_process+        request_kwargs = _request_kwargs_from_opts(opts)         spider_loader = self.crawler_process.spider_loader         if opts.spider:             try:@@ -258,7 +264,7 @@ class Command(BaseRunSpiderCommand):                 logger.error("Unable to find spider for: %(url)s", {"url": url})          async def start(spider: Spider) -> AsyncIterator[Any]:-            yield self.prepare_request(spider, Request(url), opts)+            yield self.prepare_request(spider, Request(url, **request_kwargs), opts)          if self.spidercls:             self.spidercls.start = start  # type: ignore[assignment,method-assign]scrapy/commands/shell.pyindex f64f557b5..72a547ea8 100644@@ -10,9 +10,14 @@ import asyncio from threading import Thread from typing import TYPE_CHECKING, Any, ClassVar -from scrapy.commands import ScrapyCommand+from scrapy.commands import (+    ScrapyCommand,+    _add_request_options,+    _request_kwargs_from_opts,+) from scrapy.core.engine import ExecutionEngine from scrapy.crawler import AsyncCrawlerProcess, Crawler+from scrapy.exceptions import UsageError from scrapy.http import Request from scrapy.shell import Shell from scrapy.utils.spider import DefaultSpider, spidercls_for_request@@ -59,6 +64,7 @@ class Command(ScrapyCommand):             default=False,             help="do not handle HTTP 3xx status codes and print response as-is",         )+        _add_request_options(parser)      def update_vars(self, vars: dict[str, Any]) -> None:  # noqa: A002         """You can use this function to update the Scrapy objects that will be@@ -67,6 +73,9 @@ class Command(ScrapyCommand):      def run(self, args: list[str], opts: Namespace) -> None:         url = args[0] if args else None+        request_kwargs = _request_kwargs_from_opts(opts)+        if request_kwargs and not url:+            raise UsageError("-X/--method and --data require a URL", print_help=False)         if url:             # first argument may be a local file             url = guess_scheme(url)@@ -93,7 +102,7 @@ class Command(ScrapyCommand):             self._init_without_reactor(crawler)             loop = self._get_reactorless_loop()         shell = Shell(crawler, update_vars=self.update_vars, code=opts.code, loop=loop)-        shell.start(url=url, redirect=not opts.no_redirect)+        shell.start(url=url, redirect=not opts.no_redirect, **request_kwargs)      def _init_with_reactor(self, crawler: Crawler) -> None:         # Create the engine in the main threadscrapy/shell.pyindex 019badb59..f85e7e1d1 100644@@ -137,11 +137,12 @@ class Shell:         response: Response | None = None,         spider: Spider | None = None,         redirect: bool = True,+        **kwargs: Any,     ) -> None:         # disable accidental Ctrl-C key press from shutting down the engine         signal.signal(signal.SIGINT, signal.SIG_IGN)         if url:-            self.fetch(url, spider, redirect=redirect)+            self.fetch(url, spider, redirect=redirect, **kwargs)         elif request:             self.fetch(request, spider)         elif response:tests/test_command_fetch.pyindex e25540f50..d8d0df1b4 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json from typing import TYPE_CHECKING  import pytest@@ -42,6 +43,39 @@ class TestFetchCommand:         assert "Server: TwistedWeb" in out         assert "Content-Type: text/plain" in out +    def test_data(self, mockserver: MockServer) -> None:+        returncode, out, err = proc(+            "fetch", "--data", "a=1&b=2", mockserver.url("/echo")+        )+        assert returncode == 0, err+        echo = json.loads(out)+        assert echo["body"] == "a=1&b=2"+        assert echo["headers"]["Content-Type"] == ["application/x-www-form-urlencoded"]++    def test_data_headers(self, mockserver: MockServer) -> None:+        _, out, _ = proc("fetch", "--headers", "--data", "a=1", mockserver.url("/echo"))+        out = out.replace("\r", "")  # required on win32+        assert "> Content-Type: application/x-www-form-urlencoded" in out++    @pytest.mark.parametrize("option", ["-X", "--method"])+    def test_method(self, option: str, mockserver: MockServer) -> None:+        # /echo only supports GET and POST, so a PUT gets a 501 response+        _, _, err = proc("fetch", option, "put", mockserver.url("/echo"))+        assert "downloader/request_method_count/PUT" in err+        assert "downloader/response_status_count/501" in err++    def test_method_get_with_data(self, mockserver: MockServer) -> None:+        _, out, err = proc(+            "fetch", "-X", "GET", "--data", "a=1", mockserver.url("/echo")+        )+        assert "downloader/request_method_count/GET" in err+        assert json.loads(out)["body"] == "a=1"++    def test_empty_method(self, mockserver: MockServer) -> None:+        returncode, _, err = proc("fetch", "-X", " ", mockserver.url("/echo"))+        assert returncode == 2+        assert "Invalid -X/--method value" in err+     def test_no_reactor(self, mockserver: MockServer) -> None:         _, out, _ = proc(             "fetch", "-s", "TWISTED_REACTOR_ENABLED=False", mockserver.url("/text")tests/test_command_parse.pyindex f06816b8c..23255c47b 100644@@ -119,6 +119,9 @@ class MySpider(scrapy.Spider):         else:             self.logger.debug('It Does Not Work :(') +    def parse_request_method(self, response):+        self.logger.debug(f'Request: {{response.request.method}} {{response.json()["body"]}}')+     def parse_request_without_meta(self, response):         foo = response.meta.get('foo', 'bar') @@ -306,6 +309,52 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}}         )         assert "DEBUG: It Works!" in stderr +    def test_request_with_data(self, proj_path: Path, mockserver: MockServer) -> None:+        _, _, stderr = proc(+            "parse",+            "--spider",+            self.spider_name,+            "--data",+            "a=1&b=2",+            "-c",+            "parse_request_method",+            mockserver.url("/echo"),+            cwd=proj_path,+        )+        assert "DEBUG: Request: POST a=1&b=2" in stderr++    @pytest.mark.parametrize("option", ["-X", "--method"])+    def test_request_with_method(+        self, option: str, proj_path: Path, mockserver: MockServer+    ) -> None:+        _, _, stderr = proc(+            "parse",+            "--spider",+            self.spider_name,+            option,+            "post",+            "-c",+            "parse_request_method",+            mockserver.url("/echo"),+            cwd=proj_path,+        )+        assert "DEBUG: Request: POST " in stderr++    def test_request_with_empty_method(+        self, proj_path: Path, mockserver: MockServer+    ) -> None:+        returncode, _, stderr = proc(+            "parse",+            "--spider",+            self.spider_name,+            "-X",+            "",+            mockserver.url("/echo"),+            cwd=proj_path,+        )+        assert returncode == 2+        assert "Invalid -X/--method value" in stderr+     def test_pipelines(self, proj_path: Path, mockserver: MockServer) -> None:         _, _, stderr = proc(             "parse",tests/test_command_shell.pyindex a483f5abd..9e4f81b42 100644@@ -102,6 +102,45 @@ class TestShellCommand:         assert ret == 0, out         assert "Crawled (302)" in err +    def test_data(self, mockserver: MockServer) -> None:+        code = "(request.method, request.body, response.json()['body'])"+        _, out, _ = proc(+            "shell", "--data", "a=1&b=2", mockserver.url("/echo"), "-c", code+        )+        assert out.strip() == "('POST', b'a=1&b=2', 'a=1&b=2')"++    def test_data_no_reactor(self, mockserver: MockServer) -> None:+        code = "(request.method, response.json()['body'])"+        _, out, _ = proc(+            "shell",+            "-s",+            "TWISTED_REACTOR_ENABLED=False",+            "--data",+            "a=1",+            mockserver.url("/echo"),+            "-c",+            code,+        )+        assert out.strip() == "('POST', 'a=1')"++    @pytest.mark.parametrize("option", ["-X", "--method"])+    def test_method(self, option: str, mockserver: MockServer) -> None:+        _, out, _ = proc(+            "shell",+            option,+            "put",+            mockserver.url("/echo"),+            "-c",+            "(request.method, response.status)",+        )+        assert out.strip() == "('PUT', 501)"++    @pytest.mark.parametrize("args", [("--data", "a=1"), ("-X", "POST")])+    def test_request_options_without_url(self, args: tuple[str, ...]) -> None:+        returncode, _, err = proc("shell", *args, "-c", "item")+        assert returncode == 2+        assert "-X/--method and --data require a URL" in err+     def test_request_replace(self, mockserver: MockServer) -> None:         url = mockserver.url("/text")         code = f"fetch('{url}') or fetch(response.request.replace(method='POST'))"tests/test_commands.pyindex db07a3d4d..088d6d699 100644@@ -883,6 +883,27 @@ class TestViewCommand:         body = Path(url.removeprefix("file://")).read_text(encoding="utf-8")         assert "<p class='one'>Works</p>" in body +    @pytest.mark.skipif(+        sys.platform == "win32", reason="requires a POSIX shell browser script"+    )+    def test_view_data(+        self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, mockserver: MockServer+    ) -> None:+        opened = tmp_path / "opened.txt"+        browser = tmp_path / "fake-browser.sh"+        write_recording_browser(browser, opened)+        monkeypatch.setenv("BROWSER", str(browser))++        returncode, _, err = proc(+            "view", "--data", "a=1&b=2", mockserver.url("/echo"), cwd=tmp_path+        )++        assert returncode == 0, err+        assert "downloader/request_method_count/POST" in err+        url = opened.read_text(encoding="utf-8")+        body = Path(url.removeprefix("file://")).read_text(encoding="utf-8")+        assert '"body": "a=1&b=2"' in body+     def test_non_text_response(self, mockserver: MockServer) -> None:         returncode, _, err = proc(             "view", mockserver.url("/static/files/images/scrapy.png") 
Merged fix · PR #7703+194 −21
docs/topics/commands.rstindex b9b7c2a31..1d56ee0aa 100644@@ -462,11 +462,21 @@ Supported options:  * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+  URL argument, taking the URL, HTTP method, headers, cookies and body from the+  command (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It cannot+  be combined with a URL argument.++  .. versionadded:: VERSION+ Usage examples::      $ scrapy fetch --nolog http://www.example.com/some/page.html     [ ... html content here ... ] +    $ scrapy fetch --nolog --curl 'curl -d title=hello https://httpbin.org/post'+    [ ... html content here ... ]+     $ scrapy fetch --nolog --headers http://www.example.com/     > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8     > Accept-Language: en@@ -500,6 +510,12 @@ Supported options:  * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+  URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It+  cannot be combined with a URL argument.++  .. versionadded:: VERSION+ Usage example::      $ scrapy view http://www.example.com/some/page.html@@ -528,6 +544,12 @@ Supported options:   this only affects the URL you may pass as argument on the command line;   once you are inside the shell, ``fetch(url)`` will still follow HTTP redirects by default. +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+  URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It+  cannot be combined with a URL argument.++  .. versionadded:: VERSION+ Usage example::      $ scrapy shell http://www.example.com/some/page.html@@ -566,6 +588,12 @@ Supported options: * ``--callback`` or ``-c``: spider method to use as callback for parsing the   response +* ``--curl=COMMAND``: build the request from a `cURL`_ command instead of from a+  URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It+  cannot be combined with a URL argument.++  .. versionadded:: VERSION+ * ``--meta`` or ``-m``: additional request meta that will be passed to the   callback request. This must be a valid JSON string. Example:   ``--meta='{"foo": "bar"}'``@@ -877,3 +905,5 @@ Example: .. code-block:: python      COMMANDS_MODULE = "mybot.commands"++.. _cURL: https://curl.se/scrapy/commands/__init__.pyindex c87c390c1..7805d5e2c 100644@@ -17,6 +17,7 @@ from typing import TYPE_CHECKING, Any, ClassVar from scrapy import signals from scrapy.exceptions import ScrapyDeprecationWarning, UsageError from scrapy.extensions.feedexport import FeedExporter+from scrapy.http import Request from scrapy.utils.conf import arglist_to_dict, feed_process_params_from_cli from scrapy.utils.deprecate import method_is_overridden from scrapy.utils.python import global_object_name@@ -32,6 +33,27 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__)  +def _add_curl_option(parser: argparse.ArgumentParser) -> None:+    """Register the ``--curl`` option on commands that accept a URL."""+    parser.add_argument(+        "--curl",+        metavar="COMMAND",+        default=None,+        help="build the request from a curl command, e.g. "+        "--curl 'curl -d a=1 https://example.com'; cannot be combined "+        "with a URL argument",+    )+++def _request_from_curl(curl_command: str, **kwargs: Any) -> Request:+    """Build a :class:`~scrapy.Request` from a curl command, reporting parsing+    errors as :exc:`~scrapy.exceptions.UsageError`."""+    try:+        return Request.from_curl(curl_command, **kwargs)+    except ValueError as e:+        raise UsageError(str(e), print_help=False) from e++ class _PdbHandler(logging.Handler):     """Start a post-mortem debugging session on every logged record that comes     with an exception traceback."""scrapy/commands/fetch.pyindex c345e297c..a5b44c562 100644@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, Any  from w3lib.url import is_url -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _add_curl_option, _request_from_curl from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils._datatypes import SequenceExclude@@ -34,6 +34,7 @@ class Command(ScrapyCommand):      def add_options(self, parser: ArgumentParser) -> None:         super().add_options(parser)+        _add_curl_option(parser)         parser.add_argument("--spider", dest="spider", help="use this spider")         parser.add_argument(             "--headers",@@ -67,14 +68,24 @@ class Command(ScrapyCommand):         sys.stdout.buffer.write(bytes_ + b"\n")      def run(self, args: list[str], opts: Namespace) -> None:-        if len(args) != 1 or not is_url(args[0]):-            raise UsageError-        request = Request(-            args[0],-            callback=self._print_response,-            cb_kwargs={"opts": opts},-            dont_filter=True,-        )+        if opts.curl:+            if args:+                raise UsageError("--curl cannot be combined with a URL argument")+            request = _request_from_curl(+                opts.curl,+                callback=self._print_response,+                cb_kwargs={"opts": opts},+                dont_filter=True,+            )+        else:+            if len(args) != 1 or not is_url(args[0]):+                raise UsageError+            request = Request(+                args[0],+                callback=self._print_response,+                cb_kwargs={"opts": opts},+                dont_filter=True,+            )         # by default, let the framework handle redirects,         # i.e. command handles all codes expect 3xx         if not opts.no_redirect:scrapy/commands/parse.pyindex 4edadfa58..be84af063 100644@@ -10,7 +10,7 @@ from itemadapter import ItemAdapter from twisted.internet.defer import Deferred, maybeDeferred from w3lib.url import is_url -from scrapy.commands import BaseRunSpiderCommand+from scrapy.commands import BaseRunSpiderCommand, _add_curl_option, _request_from_curl from scrapy.exceptions import UsageError from scrapy.http import Request, Response from scrapy.utils import _colorize@@ -53,6 +53,7 @@ class Command(BaseRunSpiderCommand):      def add_options(self, parser: argparse.ArgumentParser) -> None:         super().add_options(parser)+        _add_curl_option(parser)         parser.add_argument(             "--spider",             dest="spider",@@ -242,7 +243,7 @@ class Command(BaseRunSpiderCommand):             )         return None -    def set_spidercls(self, url: str, opts: argparse.Namespace) -> None:+    def set_spidercls(self, request: Request, opts: argparse.Namespace) -> None:         assert self.crawler_process         spider_loader = self.crawler_process.spider_loader         if opts.spider:@@ -253,12 +254,12 @@ class Command(BaseRunSpiderCommand):                     "Unable to find spider: %(spider)s", {"spider": opts.spider}                 )         else:-            self.spidercls = spidercls_for_request(spider_loader, Request(url))+            self.spidercls = spidercls_for_request(spider_loader, request)             if not self.spidercls:-                logger.error("Unable to find spider for: %(url)s", {"url": url})+                logger.error("Unable to find spider for: %(url)s", {"url": request.url})          async def start(spider: Spider) -> AsyncIterator[Any]:-            yield self.prepare_request(spider, Request(url), opts)+            yield self.prepare_request(spider, request, opts)          if self.spidercls:             self.spidercls.start = start  # type: ignore[assignment,method-assign]@@ -404,12 +405,18 @@ class Command(BaseRunSpiderCommand):      def run(self, args: list[str], opts: argparse.Namespace) -> None:         # parse arguments-        if not len(args) == 1 or not is_url(args[0]):-            raise UsageError-        url = args[0]+        if opts.curl:+            if args:+                raise UsageError("--curl cannot be combined with a URL argument")+            request = _request_from_curl(opts.curl)+        else:+            if not len(args) == 1 or not is_url(args[0]):+                raise UsageError+            request = Request(args[0])+        url = request.url          # prepare spidercls-        self.set_spidercls(url, opts)+        self.set_spidercls(request, opts)          if self.spidercls and opts.depth > 0:             self.start_parsing(url, opts)scrapy/commands/shell.pyindex f64f557b5..4caa43719 100644@@ -10,11 +10,13 @@ import asyncio from threading import Thread from typing import TYPE_CHECKING, Any, ClassVar -from scrapy.commands import ScrapyCommand+from scrapy.commands import ScrapyCommand, _add_curl_option, _request_from_curl from scrapy.core.engine import ExecutionEngine from scrapy.crawler import AsyncCrawlerProcess, Crawler+from scrapy.exceptions import UsageError from scrapy.http import Request from scrapy.shell import Shell+from scrapy.utils.datatypes import SequenceExclude from scrapy.utils.spider import DefaultSpider, spidercls_for_request from scrapy.utils.url import guess_scheme @@ -46,6 +48,7 @@ class Command(ScrapyCommand):      def add_options(self, parser: ArgumentParser) -> None:         super().add_options(parser)+        _add_curl_option(parser)         parser.add_argument(             "-c",             dest="code",@@ -67,7 +70,19 @@ class Command(ScrapyCommand):      def run(self, args: list[str], opts: Namespace) -> None:         url = args[0] if args else None-        if url:+        request: Request | None = None+        if opts.curl:+            if url:+                raise UsageError("--curl cannot be combined with a URL argument")+            request = _request_from_curl(opts.curl, dont_filter=True)+            if opts.no_redirect:+                request.meta["handle_httpstatus_all"] = True+            else:+                request.meta["handle_httpstatus_list"] = SequenceExclude(+                    range(300, 400)+                )+            url = request.url+        elif url:             # first argument may be a local file             url = guess_scheme(url) @@ -77,6 +92,10 @@ class Command(ScrapyCommand):         spidercls: type[Spider] = DefaultSpider         if opts.spider:             spidercls = spider_loader.load(opts.spider)+        elif request is not None:+            spidercls = spidercls_for_request(+                spider_loader, request, spidercls, log_multiple=True+            )         elif url:             spidercls = spidercls_for_request(                 spider_loader, Request(url), spidercls, log_multiple=True@@ -93,7 +112,10 @@ class Command(ScrapyCommand):             self._init_without_reactor(crawler)             loop = self._get_reactorless_loop()         shell = Shell(crawler, update_vars=self.update_vars, code=opts.code, loop=loop)-        shell.start(url=url, redirect=not opts.no_redirect)+        if request is not None:+            shell.start(request=request, redirect=not opts.no_redirect)+        else:+            shell.start(url=url, redirect=not opts.no_redirect)      def _init_with_reactor(self, crawler: Crawler) -> None:         # Create the engine in the main threadtests/test_command_fetch.pyindex e25540f50..bf239b34b 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json from typing import TYPE_CHECKING  import pytest@@ -48,6 +49,23 @@ class TestFetchCommand:         )         assert out.strip() == "Works" +    def test_curl(self, mockserver: MockServer) -> None:+        url = mockserver.url("/echo")+        _, out, _ = proc("fetch", "--curl", f"curl -d a=1 -H 'X-Test: foo' {url}")+        echo = json.loads(out)+        assert echo["body"] == "a=1"+        assert echo["headers"]["X-Test"] == ["foo"]++    def test_curl_with_url(self, mockserver: MockServer) -> None:+        url = mockserver.url("/echo")+        code, _, _ = proc("fetch", "--curl", f"curl {url}", url)+        assert code != 0++    def test_curl_invalid(self) -> None:+        code, _, err = proc("fetch", "--curl", "not-a-curl-command")+        assert code != 0+        assert "curl" in err+  class TestFetchCommandWithSpider(TestProjectBase):     @pytest.fixture(autouse=True)tests/test_command_parse.pyindex f06816b8c..663ea5799 100644@@ -239,6 +239,41 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}}         )         assert "DEBUG: It Works!" in stderr +    def test_invalid_url(self, proj_path: Path) -> None:+        code, _, _ = proc(+            "parse", "--spider", self.spider_name, "not-a-url", cwd=proj_path+        )+        assert code != 0++    def test_curl(self, proj_path: Path, mockserver: MockServer) -> None:+        _, _, stderr = proc(+            "parse",+            "--spider",+            self.spider_name,+            "-a",+            "test_arg=1",+            "--curl",+            f"curl {mockserver.url('/html')}",+            "-c",+            "parse",+            "--verbose",+            cwd=proj_path,+        )+        assert "DEBUG: It Works!" in stderr++    def test_curl_with_url(self, proj_path: Path, mockserver: MockServer) -> None:+        url = mockserver.url("/html")+        code, _, _ = proc(+            "parse",+            "--spider",+            self.spider_name,+            "--curl",+            f"curl {url}",+            url,+            cwd=proj_path,+        )+        assert code != 0+     def test_request_with_meta(self, proj_path: Path, mockserver: MockServer) -> None:         raw_json_string = '{"foo" : "baz"}'         _, _, stderr = proc(tests/test_command_shell.pyindex a483f5abd..dc156947a 100644@@ -43,6 +43,34 @@ class TestShellCommand:         _, out, _ = proc("shell", mockserver.url("/text"), "-c", "response.body")         assert "Works" in out +    def test_curl(self, mockserver: MockServer) -> None:+        url = mockserver.url("/echo")+        _, out, _ = proc(+            "shell",+            "--curl",+            f"curl -d a=1 {url}",+            "-c",+            "(request.method, request.body)",+        )+        assert "('POST', b'a=1')" in out++    def test_curl_no_redirect(self, mockserver: MockServer) -> None:+        url = mockserver.url("/redirect-no-meta-refresh")+        _, out, _ = proc(+            "shell",+            "--no-redirect",+            "--curl",+            f"curl {url}",+            "-c",+            "response.status",+        )+        assert out.strip().endswith("302")++    def test_curl_with_url(self, mockserver: MockServer) -> None:+        url = mockserver.url("/echo")+        code, _, _ = proc("shell", "--curl", f"curl {url}", url, "-c", "url")+        assert code != 0+     def test_response_type_text(self, mockserver: MockServer) -> None:         _, out, _ = proc("shell", mockserver.url("/text"), "-c", "type(response)")         assert "TextResponse" in out