Backtestsscrapyrun-20261025-211112#7025

Issue #7025 Merged fix PR #7811 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Discourage using CrawlSpider

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Patch A directly addresses the issue: it rewrites the CrawlSpider module docstring to drop the 'recommended spider' wording, adds a prominent note in docs/topics/spiders.rst saying CrawlSpider is not recommended for new code and that users should subclass Spider, adds a generic-spiders paragraph stating they are never required, discourages the `crawl` genspider template, and switches unrelated doc examples away from CrawlSpider to scrapy.Spider. It also adds a small test guarding the module docstring. The independent tests confirm it: both module-docstring tests fail on base and pass on A. Patch B never touches CrawlSpider at all; it is an unrelated refactor moving Spider attribute/method documentation into docstrings, adding type annotations, and dropping type-ignore comments. It leaves the 'recommended spider' text intact (both independent tests still fail on B) and the 2009 'most commonly used' paragraph unchanged, so it does not resolve the issue. A's tests are somewhat string-matching/superficial, but B offers nothing on the issue. A wins on correctness, tests, and scope.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

3m 54s

Wall-clock time in the container

Full test suite

No new failures

1,062 passed · 33 skipped · 1,095 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

48vs 329 merged

Changed lines, ours vs merged

I've seen a lot of user questions about CrawlSpider, most of them caused by its limitations. I'm biased but when asked I awlays reply "please consider not using it". In my experience and opinion, the control flow in a CrawlSpider spider is both confusing by itself and too different from one in a normal spider that it should be only recommended for very basic spiders with no extensibility requirements.

We currently have this text from 2009 in the CrawlSpider doc:

This is the most commonly used spider for crawling regular websites, as it provides a convenient mechanism for following links by defining a set of rules. It may not be the best suited for your particular web sites or project, but it’s generic enough for several cases, so you can start from it and override it as needed for more custom functionality, or just implement your own spider.

As a minimally needed change, I think the first half shouldn't say anything like "the most commonly used" and the second half should more prominently say that you don't need to use CrawlSpider and you should switch to Spider if you run into limitations of CrawlSpider.

I will also agree with any of the more controversial suggestions, from adding "this isn't recommended to use for new code" and up to extracting it into a separate project or even deprecating and removing.

The two patches

Our patch+41 −7
docs/topics/commands.rstindex 2a8d6779b..4a53561aa 100644@@ -271,6 +271,10 @@ Usage example::     $ scrapy genspider -t crawl scrapyorg scrapy.org     Created spider 'scrapyorg' using template 'crawl' +The ``crawl`` template creates a :class:`~scrapy.spiders.CrawlSpider`, which+is only recommended for very basic crawls; prefer the default ``basic``+template otherwise.+ This command is just a convenient shortcut for creating spiders based on pre-defined templates, but it's certainly not the only way to create spiders; you can create the spider source code files yourself instead of using thisdocs/topics/downloader-middleware.rstindex b99a7483e..f98405d8e 100644@@ -318,10 +318,10 @@ HttpAuthMiddleware      .. code-block:: python -        from scrapy.spiders import CrawlSpider+        import scrapy  -        class SomeIntranetSiteSpider(CrawlSpider):+        class SomeIntranetSiteSpider(scrapy.Spider):             name = "intranet.example.com"             custom_settings = {                 "HTTPAUTH_USER": "someuser",@@ -953,7 +953,10 @@ responses (and pass them through to your spider) you can do this:  .. code-block:: python -    class MySpider(CrawlSpider):+    import scrapy+++    class MySpider(scrapy.Spider):         handle_httpstatus_list = [301, 302]  The ``handle_httpstatus_list`` key of :attr:`Request.metadocs/topics/spider-middleware.rstindex f6bdb6383..a4cc32ac0 100644@@ -268,10 +268,10 @@ this:  .. code-block:: python -    from scrapy.spiders import CrawlSpider+    import scrapy  -    class MySpider(CrawlSpider):+    class MySpider(scrapy.Spider):         handle_httpstatus_list = [404]  .. reqmeta:: handle_httpstatus_listdocs/topics/spiders.rstindex 53f95e2bb..e95e983e3 100644@@ -445,6 +445,11 @@ your spiders from. Their aim is to provide convenient functionality for a few common scraping cases, like following all links on a site based on certain rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed. +You never need to use them: anything they do can also be done with a regular+:class:`~scrapy.Spider`, which gives you full control over the requests your+spider sends. If one of them gets in your way, switch to+:class:`~scrapy.Spider` instead of working around its limitations.+ For the examples used in the following spiders, we'll assume you have a project with a ``TestItem`` declared in a ``myproject.items`` module: @@ -465,6 +470,13 @@ with a ``TestItem`` declared in a ``myproject.items`` module: CrawlSpider ----------- +.. note:: :class:`CrawlSpider` is not recommended for new code, except for+   very basic crawls with no extensibility requirements. Its control flow is+   quite different from that of a regular :class:`~scrapy.Spider`, which makes+   it harder to understand, debug and extend. Prefer subclassing+   :class:`~scrapy.Spider` and following links from your callbacks, e.g. with+   :meth:`response.follow <scrapy.http.TextResponse.follow>`.+ .. autoclass:: CrawlSpider     :members:     :exclude-members: from_crawlerscrapy/spiders/crawl.pyindex c2fc11304..67303138c 100644@@ -1,6 +1,7 @@ """-This modules implements the CrawlSpider which is the recommended spider to use-for scraping typical websites that requires crawling pages.+This module implements the CrawlSpider, a spider that follows links based on a+set of declarative rules. It only suits basic crawls; for anything else,+subclass scrapy.Spider instead.  See documentation in docs/topics/spiders.rst """tests/test_spider_crawl.pyindex 2a0f55e61..b2716289c 100644@@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Any import pytest from w3lib.url import safe_url_string +import scrapy.spiders.crawl from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.http import HtmlResponse, Request, TextResponse from scrapy.linkextractors import LinkExtractor@@ -391,3 +392,16 @@ class TestDeprecation:     def test_crawl_spider(self):         assert issubclass(CrawlSpider, Spider)         assert isinstance(CrawlSpider(name="foo"), Spider)+++class TestDocstrings:+    def test_module_does_not_recommend_crawlspider(self):+        doc = scrapy.spiders.crawl.__doc__+        assert doc is not None+        assert "recommended spider" not in doc+        assert "scrapy.Spider" in doc++    def test_class_points_to_spider(self):+        doc = CrawlSpider.__doc__+        assert doc is not None+        assert "subclass :class:`~scrapy.Spider` instead" in doc 
Merged fix · PR #7811+177 −152
docs/topics/spiders.rstindex 53f95e2bb..0c8d1fba8 100644@@ -37,156 +37,34 @@ scrapy.Spider  .. autoclass:: scrapy.Spider -   .. attribute:: name+    .. autoattribute:: name -       A string which defines the name for this spider. The spider name is how-       the spider is located (and instantiated) by Scrapy, so it must be-       unique. However, nothing prevents you from instantiating more than one-       instance of the same spider. This is the most important spider attribute-       and it's required.+    .. autoattribute:: allowed_domains -       If the spider scrapes a single domain, a common practice is to name the-       spider after the domain, with or without the `TLD`_. So, for example, a-       spider that crawls ``mywebsite.com`` would often be called-       ``mywebsite``.+    .. autoattribute:: start_urls -   .. attribute:: allowed_domains+    .. autoattribute:: custom_settings -       An optional list of strings containing domains that this spider is-       allowed to crawl. Requests for URLs not belonging to the domain names-       specified in this list (or their subdomains) won't be followed if-       :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is-       enabled.+    .. autoattribute:: crawler -       .. versionchanged:: 2.18.0-          Changes to this attribute during a crawl are now taken into account.+    .. autoattribute:: settings -       Let's say your target URL is ``https://www.example.com/1.html``,-       then add ``'example.com'`` to the list.+    .. autoattribute:: logger -       You may modify this attribute while the spider runs, e.g. to allow-       domains that you only learn about from an earlier response. The change-       affects requests scheduled after it.+    .. autoattribute:: state -   .. autoattribute:: start_urls+    .. automethod:: update_settings -   .. attribute:: custom_settings+    .. automethod:: from_crawler -      A dictionary of settings that will be overridden from the project wide-      configuration when running this spider. It must be defined as a class-      attribute since the settings are updated before instantiation.+    .. automethod:: start -      For a list of available built-in settings see:-      :ref:`topics-settings-ref`.+    .. automethod:: parse -   .. attribute:: crawler+    .. method:: closed(reason) -      This attribute is set by the :meth:`from_crawler` class method after-      initializing the class, and links to the-      :class:`~scrapy.crawler.Crawler` object to which this spider instance is-      bound.--      Crawlers encapsulate a lot of components in the project for single-entry-      access (such as extensions, middlewares, signal managers, etc).-      See :ref:`topics-api-crawler` to know more about them.--   .. attribute:: settings--      Configuration for running this spider. This is a-      :class:`~scrapy.settings.Settings` instance, see the-      :ref:`topics-settings` topic for a detailed introduction on this subject.--   .. attribute:: logger--      Python logger created with the Spider's :attr:`name`. You can use it to-      send log messages through it as described on-      :ref:`topics-logging-from-spiders`.--   .. attribute:: state--      A dict you can use to persist some spider state between batches.-      See :ref:`topics-keeping-persistent-state-between-batches` to know more about it.--   .. method:: from_crawler(crawler, *args, **kwargs)--       This is the class method used by Scrapy to create your spiders.--       You probably won't need to override this directly because the default-       implementation acts as a proxy to the :meth:`__init__` method, calling-       it with the given arguments ``args`` and named arguments ``kwargs``.--       Nonetheless, this method sets the :attr:`crawler` and :attr:`settings`-       attributes in the new instance so they can be accessed later inside the-       spider's code.--       The settings in ``crawler.settings`` can be modified in this method,-       which is handy if you want to modify them based on arguments. As a-       consequence, these settings aren't the final values as they can be-       modified later by e.g. :ref:`add-ons <topics-addons>`. For the same-       reason, most of the :class:`~scrapy.crawler.Crawler` attributes aren't-       initialized at this point.--       The final settings and the initialized :class:`~scrapy.crawler.Crawler`-       attributes are available in the :meth:`start` method, handlers of the-       :signal:`engine_started` signal and later.--       :param crawler: crawler to which the spider will be bound-       :type crawler: :class:`~scrapy.crawler.Crawler` instance--       :param args: arguments passed to the :meth:`__init__` method-       :type args: list--       :param kwargs: keyword arguments passed to the :meth:`__init__` method-       :type kwargs: dict--   .. classmethod:: update_settings(settings)--       The ``update_settings()`` method is used to modify the spider's settings-       and is called during initialization of a spider instance.--       It takes a :class:`~scrapy.settings.Settings` object as a parameter and-       can add or update the spider's configuration values. This method is a-       class method, meaning that it is called on the :class:`~scrapy.Spider`-       class and allows all instances of the spider to share the same-       configuration.--       While per-spider settings can be set in-       :attr:`~scrapy.Spider.custom_settings`, using ``update_settings()``-       allows you to dynamically add, remove or change settings based on other-       settings, spider attributes or other factors and use setting priorities-       other than ``'spider'``. Also, it's easy to extend ``update_settings()``-       in a subclass by overriding it, while doing the same with-       :attr:`~scrapy.Spider.custom_settings` can be hard.--       For example, suppose a spider needs to modify :setting:`FEEDS`:--       .. code-block:: python--           import scrapy---           class MySpider(scrapy.Spider):-               name = "myspider"-               custom_feed = {-                   "/home/user/documents/items.json": {-                       "format": "json",-                       "indent": 4,-                   }-               }--               @classmethod-               def update_settings(cls, settings):-                   super().update_settings(settings)-                   settings.setdefault("FEEDS", {}).update(cls.custom_feed)--   .. automethod:: start--   .. automethod:: parse--   .. method:: closed(reason)--       Called when the spider closes. This method provides a shortcut to-       signals.connect() for the :signal:`spider_closed` signal.+        Called when the spider closes. This method provides a shortcut to+        signals.connect() for the :signal:`spider_closed` signal.  Let's see an example: @@ -610,7 +488,7 @@ SitemapSpider  .. autoclass:: SitemapSpider     :members:-    :exclude-members: start+    :exclude-members: from_crawler, start   SitemapSpider examples@@ -700,5 +578,4 @@ Combine SitemapSpider with other sources of urls: .. _scrapy-spider-metadata: https://scrapy-spider-metadata.readthedocs.io/en/latest/params.html .. _Sitemaps: https://www.sitemaps.org/index.html .. _robots.txt: https://www.robotstxt.org/-.. _TLD: https://en.wikipedia.org/wiki/Top-level_domain .. _Scrapyd documentation: https://scrapyd.readthedocs.io/en/latest/scrapy/extensions/spiderstate.pyindex f250dfc4c..e1cefd3f7 100644@@ -35,9 +35,9 @@ class SpiderState:     def _load_state(self, spider: Spider) -> None:         if Path(self.statefn).exists():             with Path(self.statefn).open("rb") as f:-                spider.state = pickle.load(f)  # type: ignore[attr-defined]  # noqa: S301+                spider.state = pickle.load(f)  # noqa: S301         else:-            spider.state = {}  # type: ignore[attr-defined]+            spider.state = {}      def _persist_state(self, spider: Spider) -> None:         with Path(self.statefn).open("wb") as f:@@ -49,14 +49,14 @@ class SpiderState:         assert self.crawler is not None         await self.crawler.signals.send_catch_log_async(             signals.spider_state_loaded,-            state=spider.state,  # type: ignore[attr-defined]+            state=spider.state,         )      async def _spider_closed(self, spider: Spider) -> None:         assert self.crawler is not None         await self.crawler.signals.send_catch_log_async(             signals.spider_state_saving,-            state=spider.state,  # type: ignore[attr-defined]+            state=spider.state,         )         self._persist_state(spider) @@ -71,7 +71,7 @@ class SpiderState:         assert self.crawler is not None         self.crawler.signals.send_catch_log(             signals.spider_state_loaded,-            state=spider.state,  # type: ignore[attr-defined]+            state=spider.state,         )      def spider_closed(self, spider: Spider) -> None:@@ -84,7 +84,7 @@ class SpiderState:         assert self.crawler is not None         self.crawler.signals.send_catch_log(             signals.spider_state_saving,-            state=spider.state,  # type: ignore[attr-defined]+            state=spider.state,         )         self._persist_state(spider) scrapy/spiders/__init__.pyindex 91426dbe9..fbabffde1 100644@@ -17,7 +17,7 @@ from scrapy.utils.trackref import object_ref from scrapy.utils.url import url_is_from_spider  if TYPE_CHECKING:-    from collections.abc import AsyncIterator+    from collections.abc import AsyncIterator, Collection      from twisted.internet.defer import Deferred @@ -33,13 +33,96 @@ class Spider(object_ref):     It provides a default :meth:`start` implementation that sends     requests based on the :attr:`start_urls` class attribute and calls the     :meth:`parse` method for each response.++    Like :ref:`Scrapy components <topics-components>`, spiders are+    :ref:`initialized from the crawler <from-crawler>`, through+    :meth:`~scrapy.Spider.from_crawler`, and can be :ref:`configured through+    settings <component-settings>`, which they may also override through+    :attr:`custom_settings` or :meth:`~scrapy.Spider.update_settings`.     """      name: str+    """The name of this spider.++    Every spider needs one: :class:`Spider` raises :exc:`ValueError` on+    initialization if it has no name. You usually define it as a class+    attribute, but you can also pass it at initialization time instead, e.g.+    ``CrawlerProcess.crawl(MySpider, name="myspider")`` when :ref:`running+    Scrapy from a script <run-from-script>`.++    The name is also how Scrapy locates a spider: the default :ref:`spider+    loader <topics-api-spiderloader>` indexes the spiders of your project by+    name, which is what allows the :command:`crawl` command to find them, and+    the :command:`runspider` command ignores spider classes that have no name.+    Names should hence be unique within a project; the default spider loader+    warns about duplicates and keeps only one of the matching spider classes.+    Nothing prevents you from running more than one instance of the same+    spider, though, and a custom spider loader (see+    :setting:`SPIDER_LOADER_CLASS`) may map names to spider classes in a+    completely different way.++    If the spider scrapes a single domain, a common practice is to name the+    spider after that domain, replacing dots with underscores. For example, a+    spider that crawls ``books.toscrape.com`` would often be called+    ``books_toscrape_com``.+    """+     custom_settings: dict[str, Any] | None = None+    """Settings that override the project-wide configuration when running this+    spider. It must be defined as a class attribute, since the settings are+    updated before instantiation.++    See :ref:`topics-settings-ref` for a list of built-in settings.++    .. seealso:: :meth:`~scrapy.Spider.update_settings`, a more verbose but+       more flexible alternative, which allows setting values based on other+       settings or on spider attributes, using priorities other than+       ``'spider'``, and extending the settings of a base spider class.++       :ref:`spider-settings`+    """++    allowed_domains: Collection[str]+    """The domains that this spider is allowed to crawl, if any. Requests for+    URLs not belonging to the domain names specified in this list (or their+    subdomains) won't be followed if+    :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is+    enabled.++    .. versionchanged:: 2.18.0+       Changes to this attribute during a crawl are now taken into account.++    Let's say your target URL is ``https://www.example.com/1.html``, then add+    ``'example.com'`` to the list.++    You may modify this attribute while the spider runs, e.g. to allow domains+    that you only learn about from an earlier response. The change affects+    requests scheduled after it.+    """ -    #: Start URLs. See :meth:`start`.     start_urls: list[str]+    """Start URLs. See :meth:`start`."""++    state: dict[str, Any]+    """Spider state to persist between batches. See+    :ref:`topics-keeping-persistent-state-between-batches` for details.+    """++    crawler: Crawler+    """This attribute is set by the :meth:`~scrapy.Spider.from_crawler` class+    method after initializing the class, and links to the+    :class:`~scrapy.crawler.Crawler` object to which this spider instance is+    bound.++    Crawlers encapsulate a lot of components in the project for single-entry+    access (such as extensions, middlewares, signal managers, etc). See+    :ref:`topics-api-crawler` for details.+    """++    settings: BaseSettings+    """Configuration for running this spider. See :ref:`topics-settings` for+    details.+    """      def __init__(self, name: str | None = None, **kwargs: Any):         if name is not None:@@ -52,6 +135,11 @@ class Spider(object_ref):      @property     def logger(self) -> SpiderLoggerAdapter:+        """Python logger created with the spider's :attr:`name`.++        Use it to send log messages. See :ref:`topics-logging-from-spiders` for+        details.+        """         # circular import         from scrapy.utils.log import SpiderLoggerAdapter  # noqa: PLC0415 @@ -74,13 +162,38 @@ class Spider(object_ref):      @classmethod     def from_crawler(cls, crawler: Crawler, *args: Any, **kwargs: Any) -> Self:+        """Return a new spider instance bound to *crawler*.++        You probably won't need to override this directly because the default+        implementation acts as a proxy to the ``__init__()`` method, calling+        it with the given arguments *args* and named arguments *kwargs*, which+        is how :ref:`spider arguments <spiderargs>` reach a spider.++        Nonetheless, this method sets the :attr:`crawler` and :attr:`settings`+        attributes in the new instance so they can be accessed later inside the+        spider's code.++        The settings in ``crawler.settings`` can be modified in this method,+        which is handy if you want to modify them based on arguments. As a+        consequence, these settings aren't the final values as they can be+        modified later by e.g. :ref:`add-ons <topics-addons>`. For the same+        reason, most of the :class:`~scrapy.crawler.Crawler` attributes aren't+        initialized at this point.++        The settings are final and those :class:`~scrapy.crawler.Crawler`+        attributes are initialized by the time the :meth:`start` method runs+        and the :signal:`engine_started` signal is sent, which is the earliest+        point where your spider code can rely on them.++        .. seealso:: :ref:`from-crawler`+        """         spider = cls(*args, **kwargs)         spider._set_crawler(crawler)         return spider      def _set_crawler(self, crawler: Crawler) -> None:-        self.crawler: Crawler = crawler-        self.settings: BaseSettings = crawler.settings+        self.crawler = crawler+        self.settings = crawler.settings         crawler.signals.connect(self.close, signals.spider_closed)      async def start(self) -> AsyncIterator[Any]:@@ -162,6 +275,41 @@ class Spider(object_ref):      @classmethod     def update_settings(cls, settings: BaseSettings) -> None:+        """Modify *settings*, the settings of the spider.++        This method is called during the initialization of a spider instance.+        It can add or update the spider's configuration values. It is a class+        method, meaning that it is called on the :class:`~scrapy.Spider` class+        and allows all instances of the spider to share the same configuration.++        While per-spider settings can be set in :attr:`custom_settings`, using+        this method allows you to dynamically add, remove or change settings+        based on other settings, spider attributes or other factors, and to use+        setting priorities other than ``'spider'``. Also, it's easy to extend+        this method in a subclass by overriding it, while doing the same with+        :attr:`custom_settings` can be hard.++        For example, suppose a spider needs to modify :setting:`FEEDS`:++        .. code-block:: python++            import scrapy+++            class MySpider(scrapy.Spider):+                name = "myspider"+                custom_feed = {+                    "/home/user/documents/items.json": {+                        "format": "json",+                        "indent": 4,+                    }+                }++                @classmethod+                def update_settings(cls, settings):+                    super().update_settings(settings)+                    settings.setdefault("FEEDS", {}).update(cls.custom_feed)+        """         settings.setdict(cls.custom_settings or {}, priority="spider")      @classmethodtests/test_spiderstate.pyindex bca4ad371..0a53fdb79 100644@@ -72,7 +72,7 @@ async def test_spider_state_loaded_signal_fires(tmp_path: Path) -> None:     await ss._spider_opened(spider)      assert received == [{}]-    assert spider.state == {}  # type: ignore[attr-defined]+    assert spider.state == {}   @coroutine_test@@ -82,7 +82,7 @@ async def test_spider_state_saving_signal_fires(tmp_path: Path) -> None:      spider = Spider(name="default")     await ss._spider_opened(spider)-    spider.state["key"] = "value"  # type: ignore[attr-defined]+    spider.state["key"] = "value"      saving_calls: list[dict[str, Any]] = [] tests/test_utils_url.pyindex 81b333dc2..247d38144 100644@@ -103,7 +103,7 @@ def test_url_is_from_spider_with_allowed_domains_property():         name = "myspider"          @property-        def allowed_domains(self):+        def allowed_domains(self):  # type: ignore[override]             return ["example.org"]      with pytest.warns(UserWarning, match="allowed_domains is a property"):