Backtestsscrapyrun-20261025-211112#7167

Issue #7167 Merged fix PR #7833 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Deprecate the download_delay spider attr after refactoring AutoThrottle

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches resolve the core ask: AutoThrottle stops writing `spider.download_delay` (each adds a downloader-level default delay that AutoThrottle overrides), and a user-defined `download_delay` attribute now triggers a ScrapyDeprecationWarning while still being honored. All four independent tests pass on both, so the decision rests on risk, scope, and craft.

Patch A is the conservative, focused fix. It keeps the existing precedence (spider attribute still overrides the setting, and AutoThrottle's start delay still replaces it, exactly as before), emits the warning in the same spot and style the base code already uses for `max_concurrent_requests`, adds a news entry under Deprecations, updates the settings/autothrottle/faq docs, and adds tests that cover the warning, the new `_default_slot_delay` precedence vs DOWNLOAD_SLOTS, the updated AutoThrottle unit tests, and an end-to-end mockserver crawl asserting the slot delay and that the spider attribute is no longer set.

Patch B takes a broader redesign: it deletes `_get_concurrency_delay`, bridges deprecated attributes into settings in `Crawler._apply_settings`, and makes the Downloader read only settings. That is architecturally tidy, but it carries extra risk and churn beyond the ask: (1) it changes precedence so that DOWNLOAD_DELAY set at spider or higher priority now silently wins over the attribute (previously the attribute always won), which is a behavior change folded into a deprecation step; (2) it remaps the already-deprecated `max_concurrent_requests` from CONCURRENT_REQUESTS to CONCURRENT_REQUESTS_PER_DOMAIN and rewrites a historical release-notes entry to match, which is unrelated to this issue; (3) it reads the attribute from `spidercls` when `_apply_settings` runs before spider creation, so instance-level attributes set in a spider `__init__` can be missed in flows that apply settings early, whereas the base read the instance at slot-creation time; (4) it adds no news entry for the new deprecation, removes the settings.rst note while leaving the `spider-download_delay-attribute` label dangling, and rewrites the FAQ example to DOWNLOAD_SLOTS (unrelated churn). B's tests are reasonable (bridge applied/ignored, downloader `_delay`), but it drops the AutoThrottle test matrix that covered the spider attribute as the min delay and adds no integration test.

On correctness both work; on tests A is somewhat stronger (end-to-end coverage, keeps attribute-vs-setting matrix); on scope A is tighter while B's extra changes add real risk; on craft A includes the news entry and consistent doc updates. A is the better patch.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

6m 8s

Wall-clock time in the container

Full test suite

No new failures

857 passed · 238 skipped · 1,103 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

136vs 263 merged

Changed lines, ours vs merged

We deprecated almost all spider attrs used instead of settings, but download_delay is used to set the delay calculated by AutoThrottle so it can't be deprecated right now, but it would be nice to do that when it's possible.

See the analysis/discussion at https://github.com/scrapy/scrapy/pull/7039#discussion_r2347509223

The two patches

Our patch+124 −12
docs/faq.rstindex 1a574e5da..be5bf2e1b 100644@@ -232,7 +232,9 @@ higher) in your spider:     class MySpider(CrawlSpider):         name = "myspider" -        download_delay = 2+        custom_settings = {+            "DOWNLOAD_DELAY": 2,+        }          # [ ... rest of the spider code ... ] docs/news.rstindex 8f8477eaa..2a883c37f 100644@@ -25,6 +25,15 @@ Backward-incompatible changes      (:issue:`6585`, :issue:`7731`) +Deprecations+~~~~~~~~~~~~++-   The ``download_delay`` spider attribute is deprecated, use the+    :setting:`DOWNLOAD_DELAY` setting instead.+    :class:`~scrapy.extensions.throttle.AutoThrottle` no longer sets this+    attribute on the running spider.+    (:issue:`7167`)+ .. _release-2.17.0:  Scrapy 2.17.0 (2026-07-07)docs/topics/autothrottle.rstindex 4f28019da..777649c68 100644@@ -106,8 +106,7 @@ delay of its download slot:     Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})  Note, however, that AutoThrottle still determines the starting delay of every-download slot by setting the ``download_delay`` attribute on the running-spider. If you want AutoThrottle not to impact a download slot at all, in+download slot. If you want AutoThrottle not to impact a download slot at all, in addition to setting this meta key in all requests that use that download slot, you might want to set a custom value for the ``delay`` attribute of that download slot, e.g. using :setting:`DOWNLOAD_SLOTS`.docs/topics/settings.rstindex e287c3bd5..1523b6735 100644@@ -955,7 +955,10 @@ desired.  .. note:: -    This delay can be set per spider using :attr:`download_delay` spider attribute.+    Setting this delay per spider using the ``download_delay`` spider+    attribute is deprecated. Use :attr:`Spider.custom_settings+    <scrapy.Spider.custom_settings>` or :meth:`Spider.update_settings()+    <scrapy.Spider.update_settings>` to set :setting:`DOWNLOAD_DELAY` instead.  It is possible to change this setting per domain by using :setting:`DOWNLOAD_SLOTS`.scrapy/core/downloader/__init__.pyindex eb2079d0c..39e7d581c 100644@@ -85,6 +85,7 @@ def _get_concurrency_delay( ) -> tuple[int, float]:     delay: float = settings.getfloat("DOWNLOAD_DELAY")     if hasattr(spider, "download_delay"):+        warn_on_deprecated_spider_attribute("download_delay", "DOWNLOAD_DELAY")         delay = spider.download_delay      if hasattr(spider, "max_concurrent_requests"):  # pragma: no cover@@ -120,6 +121,11 @@ class Downloader:         self.per_slot_settings: dict[str, dict[str, Any]] = self.settings.getdict(             "DOWNLOAD_SLOTS"         )+        #: Delay to use for new slots instead of the one from+        #: :setting:`DOWNLOAD_DELAY`, unless overridden in+        #: :setting:`DOWNLOAD_SLOTS`. Set by+        #: :class:`~scrapy.extensions.throttle.AutoThrottle` to its start delay.+        self._default_slot_delay: float | None = None      @inlineCallbacks     @_warn_spider_arg@@ -153,6 +159,8 @@ class Downloader:             conc, delay = _get_concurrency_delay(                 conc, self.crawler.spider, self.settings             )+            if self._default_slot_delay is not None:+                delay = self._default_slot_delay             conc, delay = (                 slot_settings.get("concurrency", conc),                 slot_settings.get("delay", delay),scrapy/extensions/throttle.pyindex 542ff1cdc..dcc912491 100644@@ -5,6 +5,7 @@ from typing import TYPE_CHECKING  from scrapy import Request, Spider, signals from scrapy.exceptions import NotConfigured+from scrapy.utils.deprecate import warn_on_deprecated_spider_attribute  if TYPE_CHECKING:     # typing.Self requires Python 3.11@@ -45,11 +46,14 @@ class AutoThrottle:     def _spider_opened(self, spider: Spider) -> None:         self.mindelay = self._min_delay(spider)         self.maxdelay = self._max_delay(spider)-        spider.download_delay = self._start_delay(spider)  # type: ignore[attr-defined]+        assert self.crawler.engine+        self.crawler.engine.downloader._default_slot_delay = self._start_delay(spider)      def _min_delay(self, spider: Spider) -> float:-        s = self.crawler.settings-        return getattr(spider, "download_delay", s.getfloat("DOWNLOAD_DELAY"))+        if hasattr(spider, "download_delay"):+            warn_on_deprecated_spider_attribute("download_delay", "DOWNLOAD_DELAY")+            return spider.download_delay+        return self.crawler.settings.getfloat("DOWNLOAD_DELAY")      def _max_delay(self, spider: Spider) -> float:         return self.crawler.settings.getfloat("AUTOTHROTTLE_MAX_DELAY")tests/test_downloaderslotssettings.pyindex 9d58a6e09..78e2769ad 100644@@ -153,3 +153,40 @@ async def test_none_slot_with_priority_queue(     stats = crawler.stats     assert stats.get_value("spider_exceptions", 0) == 0     assert stats.get_value("downloader/exception_count", 0) == 0+++@coroutine_test+async def test_deprecated_spider_download_delay():+    class _Spider(DefaultSpider):+        download_delay = 5.0++    crawler = get_crawler(_Spider, settings_dict={"DOWNLOAD_DELAY": 1.0})+    crawler.spider = crawler._create_spider()+    downloader = Downloader(crawler)+    with pytest.warns(+        ScrapyDeprecationWarning,+        match="The 'download_delay' spider attribute is deprecated",+    ):+        _, slot = downloader._get_slot(Request("https://example.com"))+    downloader.close()+    assert slot.delay == 5.0+++@coroutine_test+async def test_default_slot_delay():+    settings = {+        "DOWNLOAD_DELAY": 1.0,+        "DOWNLOAD_SLOTS": {"example.org": {"delay": 3.0}},+    }+    crawler = get_crawler(DefaultSpider, settings_dict=settings)+    crawler.spider = crawler._create_spider()+    downloader = Downloader(crawler)+    _, slot = downloader._get_slot(Request("https://example.com"))+    assert slot.delay == 1.0+    downloader._default_slot_delay = 2.0+    _, slot = downloader._get_slot(Request("https://example.net"))+    assert slot.delay == 2.0+    # DOWNLOAD_SLOTS still takes precedence.+    _, slot = downloader._get_slot(Request("https://example.org"))+    assert slot.delay == 3.0+    downloader.close()tests/test_extension_throttle.pyindex 4874f284a..b86786755 100644@@ -1,10 +1,11 @@+import warnings from logging import INFO from unittest.mock import Mock  import pytest  from scrapy import Request, Spider-from scrapy.exceptions import NotConfigured+from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning from scrapy.extensions.throttle import AutoThrottle from scrapy.http.response import Response from scrapy.settings.default_settings import (@@ -15,6 +16,8 @@ from scrapy.settings.default_settings import ( from scrapy.utils.misc import build_from_crawler from scrapy.utils.spider import DefaultSpider from scrapy.utils.test import get_crawler as _get_crawler+from tests.mockserver.http import MockServer+from tests.utils.decorators import coroutine_test  UNSET = object() @@ -22,7 +25,21 @@ UNSET = object() def get_crawler(settings=None, spidercls=None):     settings = settings or {}     settings["AUTOTHROTTLE_ENABLED"] = True-    return _get_crawler(settings_dict=settings, spidercls=spidercls)+    crawler = _get_crawler(settings_dict=settings, spidercls=spidercls)+    crawler.engine = Mock()+    crawler.engine.downloader._default_slot_delay = None+    return crawler+++def _open_spider(at, spider, spider_attr_set):+    if spider_attr_set:+        with pytest.warns(+            ScrapyDeprecationWarning,+            match="The 'download_delay' spider attribute is deprecated",+        ):+            at._spider_opened(spider)+    else:+        at._spider_opened(spider)   @pytest.mark.parametrize(@@ -82,7 +99,7 @@ def test_mindelay_definition(spider, setting, expected):      crawler = get_crawler(settings, _TestSpider)     at = build_from_crawler(AutoThrottle, crawler)-    at._spider_opened(_TestSpider())+    _open_spider(at, _TestSpider(), spider is not UNSET)     assert at.mindelay == expected  @@ -149,8 +166,13 @@ def test_startdelay_definition(min_spider, min_setting, start_setting, expected)     crawler = get_crawler(settings, _TestSpider)     at = build_from_crawler(AutoThrottle, crawler)     spider = _TestSpider()-    at._spider_opened(spider)-    assert spider.download_delay == expected+    _open_spider(at, spider, min_spider is not UNSET)+    assert crawler.engine.downloader._default_slot_delay == expected+    if min_spider is UNSET:+        assert not hasattr(spider, "download_delay")+    else:+        # AutoThrottle no longer overwrites the spider attribute.+        assert spider.download_delay == min_spider   @pytest.mark.parametrize(@@ -343,3 +365,31 @@ def test_debug_disabled(caplog):         at._response_downloaded(response, request, spider)      assert caplog.record_tuples == []+++@coroutine_test+async def test_start_delay_applied_to_slots(mockserver: MockServer) -> None:+    class _TestSpider(Spider):+        name = "test"++        async def start(self):+            yield Request(+                mockserver.url("/"), meta={"autothrottle_dont_adjust_delay": True}+            )++        def parse(self, response):+            pass++    settings = {+        "AUTOTHROTTLE_ENABLED": True,+        "AUTOTHROTTLE_START_DELAY": 0.2,+        "DOWNLOAD_DELAY": 0.1,+    }+    crawler = _get_crawler(_TestSpider, settings_dict=settings)+    with warnings.catch_warnings():+        warnings.simplefilter("error", ScrapyDeprecationWarning)+        await crawler.crawl_async()+    assert crawler.engine+    slots = crawler.engine.downloader.slots+    assert [slot.delay for slot in slots.values()] == [0.2]+    assert not hasattr(crawler.spider, "download_delay") 
Merged fix · PR #7833+132 −131
docs/faq.rstindex 1a574e5da..80658a5bf 100644@@ -220,21 +220,15 @@ the :ref:`topics-signals-ref` to know which ones. What does the response status code 999 mean? -------------------------------------------- -999 is a custom response status code used by Yahoo sites to throttle requests.+999 is a custom response status code used by some sites to throttle requests. Try slowing down the crawling speed by using a download delay of ``2`` (or-higher) in your spider:+higher) for the affected domains, with the :setting:`DOWNLOAD_SLOTS` setting:  .. code-block:: python -    from scrapy.spiders import CrawlSpider---    class MySpider(CrawlSpider):-        name = "myspider"--        download_delay = 2--        # [ ... rest of the spider code ... ]+    DOWNLOAD_SLOTS = {+        "example.com": {"delay": 2},+    }  Or by setting a global download delay in your project with the :setting:`DOWNLOAD_DELAY` setting.docs/news.rstindex 8f8477eaa..670843e0e 100644@@ -1417,7 +1417,8 @@ Deprecations      - ``download_warnsize`` (use :setting:`DOWNLOAD_WARNSIZE`) -    - ``max_concurrent_requests`` (use :setting:`CONCURRENT_REQUESTS`)+    - ``max_concurrent_requests`` (use+      :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`)      - ``user_agent`` (use :setting:`USER_AGENT`) docs/topics/autothrottle.rstindex 4f28019da..33289545c 100644@@ -106,10 +106,9 @@ delay of its download slot:     Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})  Note, however, that AutoThrottle still determines the starting delay of every-download slot by setting the ``download_delay`` attribute on the running-spider. If you want AutoThrottle not to impact a download slot at all, in-addition to setting this meta key in all requests that use that download slot,-you might want to set a custom value for the ``delay`` attribute of that+download slot. If you want AutoThrottle not to impact a download slot at all,+in addition to setting this meta key in all requests that use that download+slot, you might want to set a custom value for the ``delay`` attribute of that download slot, e.g. using :setting:`DOWNLOAD_SLOTS`.  Settingsdocs/topics/settings.rstindex e287c3bd5..65ee77258 100644@@ -953,10 +953,6 @@ desired.  .. _spider-download_delay-attribute: -.. note::--    This delay can be set per spider using :attr:`download_delay` spider attribute.- It is possible to change this setting per domain by using :setting:`DOWNLOAD_SLOTS`. extras/qpsclient.pyindex 8e5001c1d..efb582254 100644@@ -16,23 +16,20 @@ class QPSSpider(Spider):     name = "qps"     benchurl = "http://localhost:8880/" -    # Max concurrency is limited by global CONCURRENT_REQUESTS setting-    max_concurrent_requests = 8     # Requests per second goal-    qps = None  # same as: 1 / download_delay-    download_delay = None+    qps = None  # same as: 1 / DOWNLOAD_DELAY     # time in seconds to delay server responses     latency = None     # number of slots to create     slots = 1 -    def __init__(self, *a, **kw):-        super().__init__(*a, **kw)-        if self.qps is not None:-            self.qps = float(self.qps)-            self.download_delay = 1 / self.qps-        elif self.download_delay is not None:-            self.download_delay = float(self.download_delay)+    @classmethod+    def from_crawler(cls, crawler, *args, **kwargs):+        spider = super().from_crawler(crawler, *args, **kwargs)+        if spider.qps is not None:+            spider.qps = float(spider.qps)+            crawler.settings.set("DOWNLOAD_DELAY", 1 / spider.qps, priority="spider")+        return spider      async def start(self):         url = self.benchurlscrapy/core/downloader/__init__.pyindex eb2079d0c..f9ee62838 100644@@ -27,7 +27,6 @@ from scrapy.utils.defer import (     deferred_from_coro,     maybe_deferred_to_future, )-from scrapy.utils.deprecate import warn_on_deprecated_spider_attribute from scrapy.utils.httpobj import urlparse_cached  if TYPE_CHECKING:@@ -80,22 +79,6 @@ class Slot:         )  -def _get_concurrency_delay(-    concurrency: int, spider: Spider, settings: BaseSettings-) -> tuple[int, float]:-    delay: float = settings.getfloat("DOWNLOAD_DELAY")-    if hasattr(spider, "download_delay"):-        delay = spider.download_delay--    if hasattr(spider, "max_concurrent_requests"):  # pragma: no cover-        warn_on_deprecated_spider_attribute(-            "max_concurrent_requests", "CONCURRENT_REQUESTS"-        )-        concurrency = spider.max_concurrent_requests--    return concurrency, delay-- class Downloader:     DOWNLOAD_SLOT = "download_slot"     _SLOT_GC_INTERVAL: float = 60.0  # seconds@@ -112,6 +95,9 @@ class Downloader:             "CONCURRENT_REQUESTS_PER_DOMAIN"         )         self.ip_concurrency: int = self.settings.getint("CONCURRENT_REQUESTS_PER_IP")+        # Default delay of new slots. AutoThrottle overrides it to apply+        # AUTOTHROTTLE_START_DELAY.+        self._delay: float = self.settings.getfloat("DOWNLOAD_DELAY")         self.randomize_delay: bool = self.settings.getbool("RANDOMIZE_DOWNLOAD_DELAY")         self.middleware: DownloaderMiddlewareManager = (             DownloaderMiddlewareManager.from_crawler(crawler)@@ -147,16 +133,11 @@ class Downloader:     ) -> tuple[str, Slot]:         key = self.get_slot_key(request)         if key not in self.slots:-            assert self.crawler.spider             slot_settings = self.per_slot_settings.get(key, {})-            conc = self.ip_concurrency or self.domain_concurrency-            conc, delay = _get_concurrency_delay(-                conc, self.crawler.spider, self.settings-            )-            conc, delay = (-                slot_settings.get("concurrency", conc),-                slot_settings.get("delay", delay),+            conc = slot_settings.get(+                "concurrency", self.ip_concurrency or self.domain_concurrency             )+            delay = slot_settings.get("delay", self._delay)             randomize_delay = slot_settings.get("randomize_delay", self.randomize_delay)             new_slot = Slot(conc, delay, randomize_delay)             self.slots[key] = new_slotscrapy/crawler.pyindex c8d74fba7..e2f726519 100644@@ -100,6 +100,10 @@ class Crawler:             return          self.addons.load_settings(self.settings)+        self._apply_deprecated_spider_attr("download_delay", "DOWNLOAD_DELAY")+        self._apply_deprecated_spider_attr(+            "max_concurrent_requests", "CONCURRENT_REQUESTS_PER_DOMAIN"+        )         self.stats = load_object(self.settings["STATS_CLASS"])(self)          lf_cls: type[LogFormatter] = load_object(self.settings["LOG_FORMATTER"])@@ -155,6 +159,30 @@ class Crawler:             "Overridden settings:\n%(settings)s", {"settings": pprint.pformat(d)}         ) +    def _apply_deprecated_spider_attr(self, attr: str, setting: str) -> None:+        """Bridge a deprecated spider attribute onto *setting*, warning about+        the deprecation (and about being ignored when *setting* is already set+        at spider or higher priority)."""+        spider = self.spider if self.spider is not None else self.spidercls+        if not hasattr(spider, attr):+            return+        if (self.settings.getpriority(setting) or 0) >= SETTINGS_PRIORITIES["spider"]:+            warnings.warn(+                f"The {attr!r} spider attribute is deprecated. It is also being "+                f"ignored because {setting} is already set at spider or higher "+                f"priority. Remove the {attr!r} attribute from your spider.",+                category=ScrapyDeprecationWarning,+                stacklevel=3,+            )+            return+        warnings.warn(+            f"The {attr!r} spider attribute is deprecated. Use the {setting} "+            f"setting instead.",+            category=ScrapyDeprecationWarning,+            stacklevel=3,+        )+        self.settings.set(setting, getattr(spider, attr), priority="spider")+     def _apply_reactorless_default_settings(self) -> None:         """Change some setting defaults when not using a Twisted reactor. scrapy/extensions/throttle.pyindex 542ff1cdc..cde73f12e 100644@@ -43,18 +43,18 @@ class AutoThrottle:         return cls(crawler)      def _spider_opened(self, spider: Spider) -> None:-        self.mindelay = self._min_delay(spider)-        self.maxdelay = self._max_delay(spider)-        spider.download_delay = self._start_delay(spider)  # type: ignore[attr-defined]+        self.mindelay = self._min_delay()+        self.maxdelay = self._max_delay()+        assert self.crawler.engine+        self.crawler.engine.downloader._delay = self._start_delay() -    def _min_delay(self, spider: Spider) -> float:-        s = self.crawler.settings-        return getattr(spider, "download_delay", s.getfloat("DOWNLOAD_DELAY"))+    def _min_delay(self) -> float:+        return self.crawler.settings.getfloat("DOWNLOAD_DELAY") -    def _max_delay(self, spider: Spider) -> float:+    def _max_delay(self) -> float:         return self.crawler.settings.getfloat("AUTOTHROTTLE_MAX_DELAY") -    def _start_delay(self, spider: Spider) -> float:+    def _start_delay(self) -> float:         return max(             self.mindelay, self.crawler.settings.getfloat("AUTOTHROTTLE_START_DELAY")         )tests/test_crawler.pyindex b4f906e25..358f20ed7 100644@@ -74,6 +74,39 @@ class TestCrawler:         assert not settings.frozen         assert crawler.settings.frozen +    @pytest.mark.parametrize(+        ("attr", "setting"),+        [+            ("download_delay", "DOWNLOAD_DELAY"),+            ("max_concurrent_requests", "CONCURRENT_REQUESTS_PER_DOMAIN"),+        ],+    )+    def test_deprecated_spider_attr(self, attr: str, setting: str) -> None:+        crawler = get_raw_crawler(type("_Spider", (DefaultSpider,), {attr: 2}))+        with pytest.warns(+            ScrapyDeprecationWarning,+            match=f"The {attr!r} spider attribute is deprecated. Use the {setting} ",+        ):+            crawler._apply_settings()+        assert crawler.settings.getint(setting) == 2++    @pytest.mark.parametrize(+        ("attr", "setting"),+        [+            ("download_delay", "DOWNLOAD_DELAY"),+            ("max_concurrent_requests", "CONCURRENT_REQUESTS_PER_DOMAIN"),+        ],+    )+    def test_deprecated_spider_attr_ignored(self, attr: str, setting: str) -> None:+        crawler = get_raw_crawler(type("_Spider", (DefaultSpider,), {attr: 2}))+        crawler.settings.set(setting, 3, priority="spider")+        with pytest.warns(+            ScrapyDeprecationWarning,+            match=f"The {attr!r} spider attribute is deprecated. It is also being ",+        ):+            crawler._apply_settings()+        assert crawler.settings.getint(setting) == 3+     def test_crawler_accepts_dict(self) -> None:         crawler = get_crawler(DefaultSpider, {"foo": "bar"})         assert crawler.settings["foo"] == "bar"tests/test_extension_throttle.pyindex 4874f284a..2c718d95f 100644@@ -3,7 +3,7 @@ from unittest.mock import Mock  import pytest -from scrapy import Request, Spider+from scrapy import Request from scrapy.exceptions import NotConfigured from scrapy.extensions.throttle import AutoThrottle from scrapy.http.response import Response@@ -25,6 +25,13 @@ def get_crawler(settings=None, spidercls=None):     return _get_crawler(settings_dict=settings, spidercls=spidercls)  +def _mock_downloader(crawler):+    """Give *crawler* a mock engine, whose downloader AutoThrottle reads."""+    crawler.engine = Mock()+    crawler.engine.downloader.slots = {}+    return crawler.engine.downloader++ @pytest.mark.parametrize(     ("value", "expected"),     [@@ -60,29 +67,21 @@ def test_target_concurrency_invalid(value):   @pytest.mark.parametrize(-    ("spider", "setting", "expected"),+    ("setting", "expected"),     [-        (UNSET, UNSET, DOWNLOAD_DELAY),-        (1.0, UNSET, 1.0),-        (UNSET, 1.0, 1.0),-        (1.0, 2.0, 1.0),-        (3.0, 2.0, 3.0),+        (UNSET, DOWNLOAD_DELAY),+        (1.0, 1.0),     ], )-def test_mindelay_definition(spider, setting, expected):+def test_mindelay_definition(setting, expected):     settings = {}     if setting is not UNSET:         settings["DOWNLOAD_DELAY"] = setting -    class _TestSpider(Spider):-        name = "test"--    if spider is not UNSET:-        _TestSpider.download_delay = spider--    crawler = get_crawler(settings, _TestSpider)+    crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)-    at._spider_opened(_TestSpider())+    _mock_downloader(crawler)+    at._spider_opened(DefaultSpider())     assert at.mindelay == expected  @@ -99,58 +98,43 @@ def test_maxdelay_definition(value, expected):         settings["AUTOTHROTTLE_MAX_DELAY"] = value     crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)+    _mock_downloader(crawler)     at._spider_opened(DefaultSpider())     assert at.maxdelay == expected   @pytest.mark.parametrize(-    ("min_spider", "min_setting", "start_setting", "expected"),+    ("min_setting", "start_setting", "expected"),     [-        (UNSET, UNSET, UNSET, AUTOTHROTTLE_START_DELAY),-        (AUTOTHROTTLE_START_DELAY - 1.0, UNSET, UNSET, AUTOTHROTTLE_START_DELAY),-        (AUTOTHROTTLE_START_DELAY + 1.0, UNSET, UNSET, AUTOTHROTTLE_START_DELAY + 1.0),-        (UNSET, AUTOTHROTTLE_START_DELAY - 1.0, UNSET, AUTOTHROTTLE_START_DELAY),-        (UNSET, AUTOTHROTTLE_START_DELAY + 1.0, UNSET, AUTOTHROTTLE_START_DELAY + 1.0),-        (UNSET, UNSET, AUTOTHROTTLE_START_DELAY - 1.0, AUTOTHROTTLE_START_DELAY - 1.0),-        (UNSET, UNSET, AUTOTHROTTLE_START_DELAY + 1.0, AUTOTHROTTLE_START_DELAY + 1.0),-        (-            AUTOTHROTTLE_START_DELAY + 1.0,-            AUTOTHROTTLE_START_DELAY + 2.0,-            UNSET,-            AUTOTHROTTLE_START_DELAY + 1.0,-        ),+        (UNSET, UNSET, AUTOTHROTTLE_START_DELAY),+        (AUTOTHROTTLE_START_DELAY - 1.0, UNSET, AUTOTHROTTLE_START_DELAY),+        (AUTOTHROTTLE_START_DELAY + 1.0, UNSET, AUTOTHROTTLE_START_DELAY + 1.0),+        (UNSET, AUTOTHROTTLE_START_DELAY - 1.0, AUTOTHROTTLE_START_DELAY - 1.0),+        (UNSET, AUTOTHROTTLE_START_DELAY + 1.0, AUTOTHROTTLE_START_DELAY + 1.0),         (             AUTOTHROTTLE_START_DELAY + 2.0,-            UNSET,             AUTOTHROTTLE_START_DELAY + 1.0,             AUTOTHROTTLE_START_DELAY + 2.0,         ),         (             AUTOTHROTTLE_START_DELAY + 1.0,-            UNSET,             AUTOTHROTTLE_START_DELAY + 2.0,             AUTOTHROTTLE_START_DELAY + 2.0,         ),     ], )-def test_startdelay_definition(min_spider, min_setting, start_setting, expected):+def test_startdelay_definition(min_setting, start_setting, expected):     settings = {}     if min_setting is not UNSET:         settings["DOWNLOAD_DELAY"] = min_setting     if start_setting is not UNSET:         settings["AUTOTHROTTLE_START_DELAY"] = start_setting -    class _TestSpider(Spider):-        name = "test"--    if min_spider is not UNSET:-        _TestSpider.download_delay = min_spider--    crawler = get_crawler(settings, _TestSpider)+    crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)-    spider = _TestSpider()-    at._spider_opened(spider)-    assert spider.download_delay == expected+    downloader = _mock_downloader(crawler)+    at._spider_opened(DefaultSpider())+    assert downloader._delay == expected   @pytest.mark.parametrize(@@ -174,15 +158,13 @@ def test_startdelay_definition(min_spider, min_setting, start_setting, expected) def test_skipped(meta, slot):     crawler = get_crawler()     at = build_from_crawler(AutoThrottle, crawler)+    downloader = _mock_downloader(crawler)     spider = DefaultSpider()     at._spider_opened(spider)     request = Request("https://example.com", meta=meta) -    crawler.engine = Mock()-    crawler.engine.downloader = Mock()-    crawler.engine.downloader.slots = {}     if slot is not None:-        crawler.engine.downloader.slots[slot] = object()+        downloader.slots[slot] = object()     at._adjust_delay = None  # Raise exception if called.      at._response_downloaded(None, request, spider)@@ -204,18 +186,16 @@ def test_adjustment(download_latency, target_concurrency, slot_delay, expected):     settings = {"AUTOTHROTTLE_TARGET_CONCURRENCY": target_concurrency}     crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)+    downloader = _mock_downloader(crawler)     spider = DefaultSpider()     at._spider_opened(spider)     meta = {"download_latency": download_latency, "download_slot": "foo"}     request = Request("https://example.com", meta=meta)     response = Response(request.url) -    crawler.engine = Mock()-    crawler.engine.downloader = Mock()-    crawler.engine.downloader.slots = {}     slot = Mock()     slot.delay = slot_delay-    crawler.engine.downloader.slots["foo"] = slot+    downloader.slots["foo"] = slot      at._response_downloaded(response, request, spider) @@ -240,18 +220,16 @@ def test_adjustment_limits(mindelay, maxdelay, expected):     }     crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)+    downloader = _mock_downloader(crawler)     spider = DefaultSpider()     at._spider_opened(spider)     meta = {"download_latency": download_latency, "download_slot": "foo"}     request = Request("https://example.com", meta=meta)     response = Response(request.url) -    crawler.engine = Mock()-    crawler.engine.downloader = Mock()-    crawler.engine.downloader.slots = {}     slot = Mock()     slot.delay = slot_delay-    crawler.engine.downloader.slots["foo"] = slot+    downloader.slots["foo"] = slot      at._response_downloaded(response, request, spider) @@ -272,18 +250,16 @@ def test_adjustment_bad_response(     settings = {"AUTOTHROTTLE_TARGET_CONCURRENCY": target_concurrency}     crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)+    downloader = _mock_downloader(crawler)     spider = DefaultSpider()     at._spider_opened(spider)     meta = {"download_latency": download_latency, "download_slot": "foo"}     request = Request("https://example.com", meta=meta)     response = Response(request.url, status=400) -    crawler.engine = Mock()-    crawler.engine.downloader = Mock()-    crawler.engine.downloader.slots = {}     slot = Mock()     slot.delay = slot_delay-    crawler.engine.downloader.slots["foo"] = slot+    downloader.slots["foo"] = slot      at._response_downloaded(response, request, spider) @@ -294,19 +270,17 @@ def test_debug(caplog):     settings = {"AUTOTHROTTLE_DEBUG": True}     crawler = get_crawler(settings)     at = build_from_crawler(AutoThrottle, crawler)+    downloader = _mock_downloader(crawler)     spider = DefaultSpider()     at._spider_opened(spider)     meta = {"download_latency": 1.0, "download_slot": "foo"}     request = Request("https://example.com", meta=meta)     response = Response(request.url, body=b"foo") -    crawler.engine = Mock()-    crawler.engine.downloader = Mock()-    crawler.engine.downloader.slots = {}     slot = Mock()     slot.delay = 2.0     slot.transferring = (None, None)-    crawler.engine.downloader.slots["foo"] = slot+    downloader.slots["foo"] = slot      caplog.clear()     with caplog.at_level(INFO):@@ -324,19 +298,17 @@ def test_debug(caplog): def test_debug_disabled(caplog):     crawler = get_crawler()     at = build_from_crawler(AutoThrottle, crawler)+    downloader = _mock_downloader(crawler)     spider = DefaultSpider()     at._spider_opened(spider)     meta = {"download_latency": 1.0, "download_slot": "foo"}     request = Request("https://example.com", meta=meta)     response = Response(request.url, body=b"foo") -    crawler.engine = Mock()-    crawler.engine.downloader = Mock()-    crawler.engine.downloader.slots = {}     slot = Mock()     slot.delay = 2.0     slot.transferring = (None, None)-    crawler.engine.downloader.slots["foo"] = slot+    downloader.slots["foo"] = slot      caplog.clear()     with caplog.at_level(INFO):