Backtestsscrapyrun-20261025-211112#5510

Issue #5510 Merged fix PR #7885 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Support per-request download handler override

Worse

The merged fix was judged better than our patch.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches resolve the issue: the independent tests pass on both, and each lets a request pick a handler different from its scheme handler without routing other requests through it. The difference is in design coherence and completeness.

Patch A integrates the feature into the existing handler machinery: a `DOWNLOAD_HANDLERS_BY_NAME` setting registers handlers under names, the `download_handler` meta key selects by name (or by an existing scheme), lookups reuse the same `_get_handler`/`_load_handler` path (so lazy/skip_lazy, NotConfigured memoization, old-style handler detection and `_close` all work unchanged), clashes between the two settings are rejected, and named handlers are explicitly kept from becoming downloadable URL schemes. It also handles a real consequence of the feature that B ignores: the default fingerprinter now distinguishes requests that differ only by handler, so a request routed to e.g. a browser handler is not dropped as a duplicate of the same URL fetched normally. Tests cover the clash, name-in-meta, scheme-in-meta, unknown ID, the scheme/name separation, fingerprint caching after meta mutation and the robots propagation. The robots.txt inheritance is a debatable policy choice, but it is documented with an escape hatch and does not break anything.

Patch B puts a class object or import path directly in request meta and builds a parallel subsystem: three new dicts (`_meta_handlers`, `_meta_notconfigured`, `_meta_old_style_handlers`), a copy of `_load_handler`'s try/except/warn block, a copy of the old-style dispatch in `_download_with_meta_handler`, instance-reuse-by-class heuristics spliced into `_load_handler`, and identity dedup in `_close`. That duplication is the kind that drifts. It also leaves dedup unaddressed, so two requests to one URL with different handlers collide in the dupefilter. B's tests are thorough for the module (reuse, close, NotConfigured, buggy, invalid path, end-to-end crawl), which is a genuine strength, but they test a more fragile design.

On correctness both fix the issue; A additionally handles the fingerprint interaction. On scope A's setting and fingerprint changes are larger but purposeful; B's extra reuse logic adds complexity without a clear need. On craft A is cleaner and consistent with the surrounding code. A wins.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

10m 35s

Wall-clock time in the container

Full test suite

No new failures

1,474 passed · 346 skipped · 1,830 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

296vs 333 merged

Changed lines, ours vs merged

It would be great if a plugin like https://github.com/scrapy-plugins/scrapy-playwright did not had to force you to drive all requests through its download handlers, and instead you could drive certain requests through certain download handlers.

The two patches

Our patch+290 −6
docs/topics/download-handlers.rstindex b3ab386a7..212af017b 100644@@ -60,6 +60,33 @@ are honored by the default HTTP(S) handler but not necessarily by alternative ones. The same may apply to other Scrapy features, e.g. the :signal:`bytes_received` and :signal:`headers_received` signals. +.. _per-request-download-handlers:++Using a different download handler for specific requests+--------------------------------------------------------++To download a specific request with a handler different from the one+configured for its URL scheme, set the :reqmeta:`download_handler` request+meta key:++.. code-block:: python++    yield scrapy.Request(+        "https://example.com",+        meta={"download_handler": "my.download_handlers.BrowserHandler"},+    )++The value can be a download handler class or its import path. The handler+does not need to be listed in :setting:`DOWNLOAD_HANDLERS`. It is instantiated+the first time that a request needs it, and that instance is reused for later+requests. If a handler of the same class is already in use, e.g. because it is+configured for some URL scheme in :setting:`DOWNLOAD_HANDLERS`, that instance+is used instead.++If the handler cannot be instantiated, e.g. because it raises+:exc:`~scrapy.exceptions.NotConfigured`, the request is not downloaded and a+:exc:`~scrapy.exceptions.NotSupported` exception is raised.+ .. _lazy-download-handlers:  Lazy instantiation of download handlersdocs/topics/request-response.rstindex b11d2c76e..67ab53355 100644@@ -815,6 +815,7 @@ Those are: * :reqmeta:`dont_redirect` * :reqmeta:`dont_retry` * :reqmeta:`download_fail_on_dataloss`+* :reqmeta:`download_handler` * :reqmeta:`download_latency` * :reqmeta:`download_maxsize` * :reqmeta:`download_slot`@@ -917,6 +918,16 @@ download_fail_on_dataloss Whether or not to fail on broken responses. See: :setting:`DOWNLOAD_FAIL_ON_DATALOSS`. +.. reqmeta:: download_handler++download_handler+----------------++A :ref:`download handler <topics-download-handlers>` class, or its import+path, to use to download this request instead of the one configured for its+URL scheme in :setting:`DOWNLOAD_HANDLERS`. See+:ref:`per-request-download-handlers`.+ .. reqmeta:: ftp_local_filename  ftp_local_filenamescrapy/core/downloader/handlers/__init__.pyindex 57a8e962b..164bce80e 100644@@ -69,6 +69,13 @@ class DownloadHandlers:         self._notconfigured: dict[str, str] = {}         # remembers handlers with Deferred-based download_request()         self._old_style_handlers: set[str] = set()+        # stores handlers instanced for the download_handler request meta key,+        # keyed by the meta value+        self._meta_handlers: dict[str | type, DownloadHandlerProtocol] = {}+        # remembers failed handlers for the download_handler request meta key+        self._meta_notconfigured: dict[str | type, str] = {}+        # remembers meta values with Deferred-based download_request() handlers+        self._meta_old_style_handlers: set[str | type] = set()         handlers: dict[str, str | Callable[..., Any]] = without_none_values(             cast(                 "dict[str, str | Callable[..., Any]]",@@ -112,10 +119,15 @@ class DownloadHandlers:                     )                 if getattr(dhcls, "lazy", True):                     return None-            dh = build_from_crawler(-                dhcls,-                self._crawler,-            )+            # reuse an instance created for the download_handler request meta key+            for dh in self._meta_handlers.values():+                if type(dh) is dhcls:+                    break+            else:+                dh = build_from_crawler(+                    dhcls,+                    self._crawler,+                )         except NotConfigured as ex:             self._notconfigured[scheme] = str(ex)             return None@@ -149,7 +161,53 @@ class DownloadHandlers:         )         return deferred_from_coro(self.download_request_async(request)) +    def _get_meta_handler(+        self, path: str | type[DownloadHandlerProtocol]+    ) -> DownloadHandlerProtocol | None:+        """Get the handler requested through the ``download_handler`` request+        meta key, instantiating it on first use.++        If a handler of the same class is already in use, e.g. for some URL+        scheme, that instance is reused.+        """+        if path in self._meta_handlers:+            return self._meta_handlers[path]+        if path in self._meta_notconfigured:+            return None+        try:+            dhcls: type[DownloadHandlerProtocol] = load_object(path)+            for dh in (*self._handlers.values(), *self._meta_handlers.values()):+                if type(dh) is dhcls:+                    break+            else:+                dh = build_from_crawler(dhcls, self._crawler)+        except NotConfigured as ex:+            self._meta_notconfigured[path] = str(ex)+            return None+        except Exception as ex:+            logger.exception(+                'Loading "%(clspath)s" for the download_handler request meta key',+                {"clspath": path},+                extra={"crawler": self._crawler},+            )+            self._meta_notconfigured[path] = str(ex)+            return None+        self._meta_handlers[path] = dh+        if not inspect.iscoroutinefunction(dh.download_request):  # pragma: no cover+            warnings.warn(+                f"{global_object_name(dh.download_request)} is not a coroutine function."+                f" This is deprecated, please rewrite it to return a coroutine and remove"+                f" the 'spider' argument.",+                category=ScrapyDeprecationWarning,+                stacklevel=1,+            )+            self._meta_old_style_handlers.add(path)+        return dh+     async def download_request_async(self, request: Request) -> Response:+        meta_handler_path = request.meta.get("download_handler")+        if meta_handler_path is not None:+            return await self._download_with_meta_handler(request, meta_handler_path)         scheme = urlparse_cached(request).scheme         handler = self._get_handler(scheme)         if not handler:@@ -166,8 +224,33 @@ class DownloadHandlers:             )         return await handler.download_request(request) +    async def _download_with_meta_handler(+        self, request: Request, path: str | type[DownloadHandlerProtocol]+    ) -> Response:+        handler = self._get_meta_handler(path)+        if not handler:+            raise NotSupported(+                f"Unsupported download handler {path!r} set in the"+                f" download_handler request meta key:"+                f" {self._meta_notconfigured[path]}"+            )+        assert self._crawler.spider+        if path in self._meta_old_style_handlers:  # pragma: no cover+            return await maybe_deferred_to_future(+                cast(+                    "Deferred[Response]",+                    handler.download_request(request, self._crawler.spider),  # type: ignore[call-arg]+                )+            )+        return await handler.download_request(request)+     async def _close(self) -> None:-        for dh in self._handlers.values():+        handlers: list[DownloadHandlerProtocol] = []+        for dh in (*self._handlers.values(), *self._meta_handlers.values()):+            # the same instance can be used by several schemes or meta values+            if not any(dh is h for h in handlers):+                handlers.append(dh)+        for dh in handlers:             if not hasattr(dh, "close"):  # pragma: no cover                 warnings.warn(                     f"{global_object_name(dh)} doesn't define a close() method."tests/test_downloader_handlers.pyindex ad77fd7dd..3421ecb6f 100644@@ -17,6 +17,7 @@ from twisted.web.client import ResponseFailed from twisted.web.error import SchemeNotSupported from w3lib.url import path_to_file_uri +from scrapy import signals from scrapy.core.downloader.handlers import DownloadHandlers from scrapy.core.downloader.handlers.datauri import DataURIDownloadHandler from scrapy.core.downloader.handlers.file import FileDownloadHandler@@ -28,11 +29,13 @@ from scrapy.exceptions import (     DownloadFailedError,     DownloadTimeoutError,     NotConfigured,+    NotSupported,     ScrapyDeprecationWarning,     UnsupportedURLSchemeError, )-from scrapy.http import Request, TextResponse+from scrapy.http import Request, Response, TextResponse from scrapy.responsetypes import responsetypes+from scrapy.spiders import Spider from scrapy.utils._download_handlers import wrap_twisted_exceptions from scrapy.utils.boto import is_botocore_available from scrapy.utils.misc import build_from_crawler@@ -131,6 +134,166 @@ class TestLoad:         assert "scheme" not in dh._notconfigured  +class OverrideDH:+    lazy = False+    instances: list[OverrideDH] = []++    def __init__(self) -> None:+        self.closed = False+        OverrideDH.instances.append(self)++    async def download_request(self, request: Request) -> Response:+        return Response(request.url, body=b"override", request=request)++    async def close(self) -> None:+        self.closed = True+++class LazyOverrideDH(OverrideDH):+    lazy = True+++class OverrideSpider(Spider):+    name = "override"++    async def start(self):+        yield Request(+            "https://example.com/a",+            meta={"download_handler": OverrideDH},+        )+        yield Request(+            "https://example.com/b",+            meta={"download_handler": OffDH},+            errback=self.errback,+        )+        yield Request("data:,default")++    def parse(self, response):+        yield {"url": response.url, "body": response.body}++    def errback(self, failure):+        yield {"url": failure.request.url, "error": failure.type.__name__}+++class TestPerRequestHandler:+    def setup_method(self):+        OverrideDH.instances = []++    def _get_dh(self, settings=None) -> DownloadHandlers:+        crawler = get_crawler(settings_dict=settings or {})+        crawler.spider = crawler._create_spider("foo")+        return DownloadHandlers(crawler)++    @coroutine_test+    async def test_meta_class(self):+        dh = self._get_dh()+        request = Request("data:,default", meta={"download_handler": OverrideDH})+        response = await dh.download_request_async(request)+        assert response.body == b"override"+        # requests without the meta key still use the scheme handler+        response = await dh.download_request_async(Request("data:,default"))+        assert response.body == b"default"++    @coroutine_test+    async def test_meta_import_path(self):+        dh = self._get_dh()+        path = "tests.test_downloader_handlers.OverrideDH"+        for _ in range(2):+            request = Request("https://example.com", meta={"download_handler": path})+            response = await dh.download_request_async(request)+            assert response.body == b"override"+        # instantiated once and reused+        assert len(OverrideDH.instances) == 1++    @coroutine_test+    async def test_meta_reuses_scheme_instance(self):+        dh = self._get_dh({"DOWNLOAD_HANDLERS": {"custom": OverrideDH}})+        assert len(OverrideDH.instances) == 1+        request = Request("data:,default", meta={"download_handler": OverrideDH})+        response = await dh.download_request_async(request)+        assert response.body == b"override"+        assert len(OverrideDH.instances) == 1+        await dh._close()+        assert OverrideDH.instances[0].closed++    @coroutine_test+    async def test_lazy_scheme_reuses_meta_instance(self):+        dh = self._get_dh({"DOWNLOAD_HANDLERS": {"custom": LazyOverrideDH}})+        assert len(OverrideDH.instances) == 0+        request = Request("data:,default", meta={"download_handler": LazyOverrideDH})+        response = await dh.download_request_async(request)+        assert response.body == b"override"+        response = await dh.download_request_async(Request("custom://example"))+        assert response.body == b"override"+        assert len(OverrideDH.instances) == 1++    @coroutine_test+    async def test_meta_handler_closed(self):+        dh = self._get_dh()+        request = Request("data:,", meta={"download_handler": OverrideDH})+        await dh.download_request_async(request)+        assert not OverrideDH.instances[0].closed+        await dh._close()+        assert OverrideDH.instances[0].closed++    @coroutine_test+    async def test_meta_not_configured(self):+        dh = self._get_dh()+        request = Request("data:,", meta={"download_handler": OffDH})+        with pytest.raises(NotSupported, match="Unsupported download handler"):+            await dh.download_request_async(request)+        # the failure is remembered+        assert OffDH in dh._meta_notconfigured+        with pytest.raises(NotSupported, match="Unsupported download handler"):+            await dh.download_request_async(request)++    @coroutine_test+    async def test_meta_buggy(self, caplog: pytest.LogCaptureFixture) -> None:+        dh = self._get_dh()+        request = Request("data:,", meta={"download_handler": BuggyDH})+        with pytest.raises(NotSupported, match="Unsupported download handler"):+            await dh.download_request_async(request)+        assert "for the download_handler request meta key" in caplog.text++    @coroutine_test+    async def test_meta_invalid_path(self) -> None:+        dh = self._get_dh()+        request = Request(+            "data:,", meta={"download_handler": "tests.nonexistent.Handler"}+        )+        with pytest.raises(NotSupported, match="tests.nonexistent.Handler"):+            await dh.download_request_async(request)++    @coroutine_test+    async def test_meta_unsupported_scheme(self):+        # the override works even for schemes without a configured handler+        dh = self._get_dh({"DOWNLOAD_HANDLERS": {"https": None}})+        request = Request("https://example.com", meta={"download_handler": OverrideDH})+        response = await dh.download_request_async(request)+        assert response.body == b"override"++    @coroutine_test+    async def test_crawl(self):+        items = []++        def on_item(item):+            items.append(item)++        crawler = get_crawler(+            OverrideSpider,+            # make sure no real HTTPS download can happen+            {"DOWNLOAD_HANDLERS": {"https": None}},+        )+        crawler.signals.connect(on_item, signal=signals.item_scraped)+        await crawler.crawl_async()+        by_url = {item["url"]: item for item in items}+        assert by_url["https://example.com/a"]["body"] == b"override"+        assert by_url["https://example.com/b"]["error"] == "NotSupported"+        assert by_url["data:,default"]["body"] == b"default"+        assert len(OverrideDH.instances) == 1+        assert OverrideDH.instances[0].closed++ class TestFile:     def setup_method(self):         # add a special char to check that they are handled correctly 
Merged fix · PR #7885+291 −42
docs/topics/download-handlers.rstindex b3ab386a7..19e5de2e3 100644@@ -43,6 +43,48 @@ for additional schemes and to replace or disable default ones:     :ref:`security-local-resources`, for the security implications of the     default ``http``, ``ftp``, ``file`` and ``data`` handlers. +.. _download-handler-ids:++Choosing a download handler per request+---------------------------------------++.. versionadded:: VERSION++The :setting:`DOWNLOAD_HANDLERS_BY_NAME` setting registers handlers under a+name instead of a URL scheme. Such handlers are only used by requests that ask+for them through the :reqmeta:`download_handler` metadata key:++.. code-block:: python++    DOWNLOAD_HANDLERS_BY_NAME = {+        "playwright": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",+    }++.. invisible-code-block: python++    from scrapy import Request++.. code-block:: python++    Request("https://example.com", meta={"download_handler": "playwright"})++:reqmeta:`download_handler` also accepts a URL scheme, e.g. ``"https"``, to use+a handler from :setting:`DOWNLOAD_HANDLERS` on a request whose URL uses a+different scheme.++Requests keep their :reqmeta:`download_handler` across redirects and retries,+and :ref:`robots.txt <topics-dlmw-robots>` requests inherit it from the request+that triggers them. To fetch ``robots.txt`` differently, use a :ref:`downloader+middleware <topics-downloader-middleware>` that acts on requests with the+:reqmeta:`is_robotstxt_request` metadata key:++.. code-block:: python++    class DirectRobotsTxtMiddleware:+        def process_request(self, request):+            if request.meta.get("is_robotstxt_request"):+                request.meta.pop("download_handler", None)+ Replacing HTTP(S) download handlers ----------------------------------- docs/topics/request-response.rstindex b11d2c76e..c606cc02b 100644@@ -889,6 +889,17 @@ This meta key is not supported by :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`, but the :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported by them. +.. reqmeta:: download_handler++download_handler+----------------++.. versionadded:: VERSION++ID of the :ref:`download handler <topics-download-handlers>` to use for this+request, either a name from :setting:`DOWNLOAD_HANDLERS_BY_NAME` or a URL+scheme from :setting:`DOWNLOAD_HANDLERS`. See :ref:`download-handler-ids`.+ .. reqmeta:: download_timeout  download_timeout@@ -963,6 +974,18 @@ http_user  Overrides :setting:`HTTPAUTH_USER` for this request. +.. reqmeta:: is_robotstxt_request++is_robotstxt_request+--------------------++.. versionadded:: VERSION++``True`` on the ``robots.txt`` requests that+:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` sends, so+that :ref:`downloader middlewares <topics-downloader-middleware>` can tell them+apart from other requests.+ .. reqmeta:: max_retry_times  max_retry_timesdocs/topics/settings.rstindex c396fb1c1..afa587fb3 100644@@ -1145,6 +1145,19 @@ handler (without replacement), place this in your ``settings.py``:     :ref:`security-local-resources`  +.. setting:: DOWNLOAD_HANDLERS_BY_NAME++DOWNLOAD_HANDLERS_BY_NAME+-------------------------++.. versionadded:: VERSION++Default: ``{}``++A dict mapping names to :ref:`download handlers <topics-download-handlers>`+that requests can ask for by name. See :ref:`download-handler-ids`.++ .. reqmeta:: download_slot .. setting:: DOWNLOAD_SLOTS scrapy/core/downloader/handlers/__init__.pyindex 57a8e962b..68db7ed49 100644@@ -61,44 +61,59 @@ class DownloadHandlerProtocol(Protocol): class DownloadHandlers:     def __init__(self, crawler: Crawler):         self._crawler: Crawler = crawler-        # stores acceptable schemes on instancing-        self._schemes: dict[str, str | Callable[..., Any]] = {}-        # stores instanced handlers for schemes+        # stores class paths by handler ID+        self._paths: dict[str, str | Callable[..., Any]] = {}+        # handler IDs that are also acceptable URL schemes+        self._schemes: set[str] = set()+        # stores instanced handlers by handler ID         self._handlers: dict[str, DownloadHandlerProtocol] = {}         # remembers failed handlers         self._notconfigured: dict[str, str] = {}         # remembers handlers with Deferred-based download_request()         self._old_style_handlers: set[str] = set()-        handlers: dict[str, str | Callable[..., Any]] = without_none_values(+        scheme_handlers: dict[str, str | Callable[..., Any]] = without_none_values(             cast(                 "dict[str, str | Callable[..., Any]]",                 crawler.settings.getwithbase("DOWNLOAD_HANDLERS"),             )         )-        for scheme, clspath in handlers.items():-            self._schemes[scheme] = clspath-            self._load_handler(scheme, skip_lazy=True)+        named_handlers: dict[str, str | Callable[..., Any]] = without_none_values(+            cast(+                "dict[str, str | Callable[..., Any]]",+                crawler.settings.getdict("DOWNLOAD_HANDLERS_BY_NAME"),+            )+        )+        if clashes := scheme_handlers.keys() & named_handlers.keys():+            raise ValueError(+                f"The following download handler IDs are defined both in "+                f"DOWNLOAD_HANDLERS and in DOWNLOAD_HANDLERS_BY_NAME: "+                f"{', '.join(sorted(clashes))}."+            )+        self._schemes.update(scheme_handlers)+        for handler_id, clspath in (scheme_handlers | named_handlers).items():+            self._paths[handler_id] = clspath+            self._load_handler(handler_id, skip_lazy=True)          crawler.signals.connect(self._close, signals.engine_stopped) -    def _get_handler(self, scheme: str) -> DownloadHandlerProtocol | None:-        """Lazy-load the downloadhandler for a scheme-        only on the first request for that scheme.+    def _get_handler(self, handler_id: str) -> DownloadHandlerProtocol | None:+        """Lazy-load the download handler with the given ID only on its first+        request.         """-        if scheme in self._handlers:-            return self._handlers[scheme]-        if scheme in self._notconfigured:+        if handler_id in self._handlers:+            return self._handlers[handler_id]+        if handler_id in self._notconfigured:             return None-        if scheme not in self._schemes:-            self._notconfigured[scheme] = "no handler available for that scheme"+        if handler_id not in self._paths:+            self._notconfigured[handler_id] = "no handler with that ID"             return None -        return self._load_handler(scheme)+        return self._load_handler(handler_id)      def _load_handler(-        self, scheme: str, skip_lazy: bool = False+        self, handler_id: str, skip_lazy: bool = False     ) -> DownloadHandlerProtocol | None:-        path = self._schemes[scheme]+        path = self._paths[handler_id]         try:             dhcls: type[DownloadHandlerProtocol] = load_object(path)             if skip_lazy:@@ -117,17 +132,17 @@ class DownloadHandlers:                 self._crawler,             )         except NotConfigured as ex:-            self._notconfigured[scheme] = str(ex)+            self._notconfigured[handler_id] = str(ex)             return None         except Exception as ex:             logger.exception(-                'Loading "%(clspath)s" for scheme "%(scheme)s"',-                {"clspath": path, "scheme": scheme},+                'Loading "%(clspath)s" for handler ID "%(handler_id)s"',+                {"clspath": path, "handler_id": handler_id},                 extra={"crawler": self._crawler},             )-            self._notconfigured[scheme] = str(ex)+            self._notconfigured[handler_id] = str(ex)             return None-        self._handlers[scheme] = dh+        self._handlers[handler_id] = dh         if not inspect.iscoroutinefunction(dh.download_request):  # pragma: no cover             warnings.warn(                 f"{global_object_name(dh.download_request)} is not a coroutine function."@@ -136,7 +151,7 @@ class DownloadHandlers:                 category=ScrapyDeprecationWarning,                 stacklevel=1,             )-            self._old_style_handlers.add(scheme)+            self._old_style_handlers.add(handler_id)         return dh      def download_request(@@ -150,14 +165,33 @@ class DownloadHandlers:         return deferred_from_coro(self.download_request_async(request))      async def download_request_async(self, request: Request) -> Response:-        scheme = urlparse_cached(request).scheme-        handler = self._get_handler(scheme)-        if not handler:-            raise NotSupported(-                f"Unsupported URL scheme '{scheme}': {self._notconfigured[scheme]}"-            )+        handler_id = request.meta.get("download_handler")+        if handler_id is None:+            scheme = urlparse_cached(request).scheme+            # Handler IDs from DOWNLOAD_HANDLERS_BY_NAME are only reachable+            # through the download_handler request metadata key, so that+            # registering one does not make a new URL scheme downloadable.+            if scheme in self._schemes:+                handler = self._get_handler(scheme)+            else:+                handler = None+                self._notconfigured.setdefault(+                    scheme, "no handler available for that scheme"+                )+            if not handler:+                raise NotSupported(+                    f"Unsupported URL scheme '{scheme}': {self._notconfigured[scheme]}"+                )+            handler_id = scheme+        else:+            handler = self._get_handler(handler_id)+            if not handler:+                raise NotSupported(+                    f"Unusable download handler {handler_id!r}: "+                    f"{self._notconfigured[handler_id]}"+                )         assert self._crawler.spider-        if scheme in self._old_style_handlers:  # pragma: no cover+        if handler_id in self._old_style_handlers:  # pragma: no cover             return await maybe_deferred_to_future(                 cast(                     "Deferred[Response]",scrapy/downloadermiddlewares/robotstxt.pyindex 38659024e..210d52586 100644@@ -7,7 +7,7 @@ enable this middleware and enable the ROBOTSTXT_OBEY setting. from __future__ import annotations  import logging-from typing import TYPE_CHECKING, Self+from typing import TYPE_CHECKING, Any, Self  from twisted.internet.defer import Deferred @@ -87,10 +87,16 @@ class RobotsTxtMiddleware:         if netloc not in self._parsers:             self._parsers[netloc] = Deferred()             robotsurl = f"{url.scheme}://{url.netloc}/robots.txt"+            meta: dict[str, Any] = {+                "dont_obey_robotstxt": True,+                "is_robotstxt_request": True,+            }+            if "download_handler" in request.meta:+                meta["download_handler"] = request.meta["download_handler"]             robotsreq = Request(                 robotsurl,                 priority=self.DOWNLOAD_PRIORITY,-                meta={"dont_obey_robotstxt": True},+                meta=meta,                 callback=NO_CALLBACK,             )             try:scrapy/settings/default_settings.pyindex c215e23ce..62a32acf4 100644@@ -73,6 +73,7 @@ __all__ = [     "DOWNLOAD_FAIL_ON_DATALOSS",     "DOWNLOAD_HANDLERS",     "DOWNLOAD_HANDLERS_BASE",+    "DOWNLOAD_HANDLERS_BY_NAME",     "DOWNLOAD_MAXSIZE",     "DOWNLOAD_SLOTS",     "DOWNLOAD_TIMEOUT",@@ -310,6 +311,7 @@ DOWNLOAD_HANDLERS_BASE = {     "s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",     "ftp": "scrapy.core.downloader.handlers.ftp.FTPDownloadHandler", }+DOWNLOAD_HANDLERS_BY_NAME: dict[str, str] = {}  DOWNLOAD_MAXSIZE = 1024 * 1024 * 1024  # 1024m DOWNLOAD_WARNSIZE = 32 * 1024 * 1024  # 32mscrapy/utils/request.pyindex 6a75f8193..248b4cd28 100644@@ -29,7 +29,7 @@ logger = logging.getLogger(__name__)   _fingerprint_cache: WeakKeyDictionary[-    Request, dict[tuple[tuple[bytes, ...] | None, bool, bool], bytes]+    Request, dict[tuple[tuple[bytes, ...] | None, bool, bool, str | None], bytes] ] = WeakKeyDictionary()  @@ -78,8 +78,18 @@ def fingerprint(         )     verbatim_url = bool(request.meta.get("verbatim_url"))     effective_keep_fragments = keep_fragments and not verbatim_url+    # A handler ID matching the URL scheme is the one that would be used+    # anyway, so it is left out to keep such fingerprints unchanged.+    handler_id: str | None = request.meta.get("download_handler")+    if handler_id is not None and handler_id == urlparse_cached(request).scheme:+        handler_id = None     cache = _fingerprint_cache.setdefault(request, {})-    cache_key = (processed_include_headers, effective_keep_fragments, verbatim_url)+    cache_key = (+        processed_include_headers,+        effective_keep_fragments,+        verbatim_url,+        handler_id,+    )     if cache_key not in cache:         # To decode bytes reliably (JSON does not support bytes), regardless of         # character encoding, we use bytes.hex()@@ -101,6 +111,8 @@ def fingerprint(             "body": (request.body or b"").hex(),             "headers": headers,         }+        if handler_id is not None:+            fingerprint_data["download_handler"] = handler_id         fingerprint_json = json.dumps(fingerprint_data, sort_keys=True)         cache[cache_key] = hashlib.sha1(  # noqa: S324             fingerprint_json.encode()@@ -115,13 +127,17 @@ class RequestFingerprinterProtocol(Protocol): class RequestFingerprinter:     """Default fingerprinter. +    .. versionchanged:: VERSION+       :reqmeta:`download_handler` is taken into account.+     It takes into account a canonical version     (:func:`w3lib.url.canonicalize_url`) of :attr:`request.url     <scrapy.Request.url>` and the values of :attr:`request.method     <scrapy.Request.method>` and :attr:`request.body     <scrapy.Request.body>`, unless :reqmeta:`verbatim_url` is true for that-    request. It then generates an `SHA1 <https://en.wikipedia.org/wiki/SHA-1>`_-    hash.+    request. It also takes into account :reqmeta:`download_handler` when it+    does not match the URL scheme. It then generates an `SHA1+    <https://en.wikipedia.org/wiki/SHA-1>`_ hash.     """      @classmethodtests/test_downloader_handlers.pyindex ad77fd7dd..ca9860271 100644@@ -28,10 +28,11 @@ from scrapy.exceptions import (     DownloadFailedError,     DownloadTimeoutError,     NotConfigured,+    NotSupported,     ScrapyDeprecationWarning,     UnsupportedURLSchemeError, )-from scrapy.http import Request, TextResponse+from scrapy.http import Request, Response, TextResponse from scrapy.responsetypes import responsetypes from scrapy.utils._download_handlers import wrap_twisted_exceptions from scrapy.utils.boto import is_botocore_available@@ -47,6 +48,20 @@ class DummyDH:         pass  +class NamedDH:+    lazy = False++    async def download_request(self, request):+        return Response(request.url, body=b"named")+++class SchemeDH:+    lazy = False++    async def download_request(self, request):+        return Response(request.url, body=b"scheme")++ class DummyLazyDH:     # Default (but deprecated) is lazy for backward compatibility     async def download_request(self, request):@@ -100,7 +115,7 @@ class TestLoad:         assert "scheme" not in dh._handlers         assert "scheme" in dh._notconfigured         assert (-            'Loading "<class \'tests.test_downloader_handlers.BuggyDH\'>" for scheme "scheme"'+            'Loading "<class \'tests.test_downloader_handlers.BuggyDH\'>" for handler ID "scheme"'             in caplog.text         ) @@ -131,6 +146,58 @@ class TestLoad:         assert "scheme" not in dh._notconfigured  +class TestHandlerID:+    @staticmethod+    def _get_dh() -> DownloadHandlers:+        crawler = get_crawler(+            settings_dict={+                "DOWNLOAD_HANDLERS": {"https": SchemeDH},+                "DOWNLOAD_HANDLERS_BY_NAME": {"named": NamedDH},+            }+        )+        crawler.spider = crawler._create_spider()+        return DownloadHandlers(crawler)++    def test_clashing_ids(self) -> None:+        crawler = get_crawler(+            settings_dict={+                "DOWNLOAD_HANDLERS": {"https": SchemeDH},+                "DOWNLOAD_HANDLERS_BY_NAME": {"https": NamedDH},+            }+        )+        with pytest.raises(ValueError, match="defined both in DOWNLOAD_HANDLERS"):+            DownloadHandlers(crawler)++    @coroutine_test+    async def test_name_not_a_scheme(self) -> None:+        dh = self._get_dh()+        with pytest.raises(NotSupported, match="Unsupported URL scheme 'named'"):+            await dh.download_request_async(Request("named://example.com"))++    @coroutine_test+    async def test_name_in_meta(self) -> None:+        dh = self._get_dh()+        request = Request("https://example.com", meta={"download_handler": "named"})+        response = await dh.download_request_async(request)+        assert response.body == b"named"++    @coroutine_test+    async def test_scheme_in_meta(self) -> None:+        dh = self._get_dh()+        request = Request("ftp://example.com", meta={"download_handler": "https"})+        response = await dh.download_request_async(request)+        assert response.body == b"scheme"++    @coroutine_test+    async def test_unknown_id_in_meta(self) -> None:+        dh = self._get_dh()+        request = Request("https://example.com", meta={"download_handler": "nmaed"})+        with pytest.raises(+            NotSupported, match="Unusable download handler 'nmaed': no handler with"+        ):+            await dh.download_request_async(request)++ class TestFile:     def setup_method(self):         # add a special char to check that they are handled correctlytests/test_downloadermiddleware_robotstxt.pyindex f55e57b4f..62b91e9f4 100644@@ -275,6 +275,17 @@ Disallow: /some/randome/page.html         await middleware.process_request(Request("http://site.local/allowed"))         assert middleware.process_request_2.called +    @coroutine_test+    async def test_robotstxt_download_handler(self) -> None:+        middleware = RobotsTxtMiddleware(self._get_successful_crawler())+        await self.assertNotIgnored(+            Request("http://site.local/allowed", meta={"download_handler": "named"}),+            middleware,+        )+        robotsreq = self.crawler.engine.download_async.call_args_list[0][0][0]+        assert robotsreq.meta["download_handler"] == "named"+        assert robotsreq.meta["is_robotstxt_request"] is True+     async def assertNotIgnored(         self, request: Request, middleware: RobotsTxtMiddleware     ) -> None:tests/test_utils_request.pyindex 32130ded8..ce0ac8ff5 100644@@ -77,13 +77,15 @@ class TestFingerprint:     function: _FingerprintFunction = staticmethod(fingerprint)     cache: (         WeakKeyDictionary[-            Request, dict[tuple[tuple[bytes, ...] | None, bool, bool], bytes]+            Request,+            dict[tuple[tuple[bytes, ...] | None, bool, bool, str | None], bytes],         ]         | WeakKeyDictionary[-            Request, dict[tuple[tuple[bytes, ...] | None, bool, bool], str]+            Request,+            dict[tuple[tuple[bytes, ...] | None, bool, bool, str | None], str],         ]     ) = _fingerprint_cache-    default_cache_key = (None, False, False)+    default_cache_key = (None, False, False, None)     known_hashes: tuple[tuple[Request, bytes | str, dict[str, Any]], ...] = (         (             Request("http://example.org"),@@ -239,6 +241,39 @@ class TestFingerprint:         assert self.function(r5) == self.function(r5, keep_fragments=True)         assert self.function(r5) == self.function(r5, keep_fragments=False) +    def test_download_handler(self):+        r1 = Request("https://example.com")+        r2 = Request("https://example.com", meta={"download_handler": "playwright"})+        r3 = Request("https://example.com", meta={"download_handler": "http"})+        assert self.function(r1) != self.function(r2)+        assert self.function(r2) != self.function(r3)++        # An ID matching the URL scheme is the handler that would be used+        # anyway, so it does not change the fingerprint.+        r4 = Request("https://example.com", meta={"download_handler": "https"})+        assert self.function(r1) == self.function(r4)++        # IDs are case-sensitive, and URL schemes are lowercased.+        r5 = Request("HTTPS://example.com", meta={"download_handler": "HTTPS"})+        assert self.function(r1) != self.function(r5)++    def test_download_handler_caching(self):+        # The cached fingerprint must account for the handler ID, which can+        # change after the first call.+        r1 = Request("https://example.com")+        fp1 = self.function(r1)+        r1.meta["download_handler"] = "playwright"+        assert self.function(r1) != fp1++    def test_download_handler_and_verbatim_url(self):+        r1 = Request(+            "https://example.com/a b",+            meta={"verbatim_url": True, "download_handler": "playwright"},+        )+        r2 = Request("https://example.com/a b", meta={"verbatim_url": True})+        r3 = Request("https://example.com/a b", meta={"download_handler": "playwright"})+        assert len({self.function(r1), self.function(r2), self.function(r3)}) == 3+     def test_method_and_body(self):         r1 = Request("http://www.example.com")         r2 = Request("http://www.example.com", method="POST")