Backtestsscrapyrun-20261025-211112#2378

Issue #2378 Merged fix PR #8051 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

get_project_settings of doesn't reflect per-spider settings

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

The issue is a user misunderstanding with two components: get_project_settings() is documented nowhere and does not include spider settings, and custom_settings assigned in __init__ is silently ignored because spider settings are read from the class before instantiation. Patch A only adds a docstring and a brief docs mention of get_project_settings; it does nothing about the silent-ignore half, which is what actually confused the reporter, and adds no tests. Patch B documents both points explicitly (a note in settings.rst stating custom_settings must be a class attribute and that get_project_settings excludes spider settings, plus a fuller docstring), and adds a targeted runtime UserWarning in Crawler._create_spider when custom_settings is found on the instance and differs from the class attribute. The check is narrow and low-risk (vars() membership plus identity comparison), covers both the __init__ case and the spider-argument case, and points users at the correct alternatives (class attribute, update_settings, from_crawler with priority='spider'). It is backed by four tests that would fail before the change and pass after, including a full crawl_async run. The independent evidence confirms this: only Patch B passes the 'not silently ignored' test, while the 'override in __init__' test fails on both since neither (correctly) changes the settings model. The warning is a modest behavior addition beyond pure docs, but it directly targets the reported confusion rather than being unrelated churn. B is the more complete and better-verified fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

10m 36s

Wall-clock time in the container

Full test suite

No new failures

808 passed · 268 skipped · 1,082 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

107vs 12 merged

Changed lines, ours vs merged

It's possible I'm using this wrong. But I'm doing the following:

class MySpider (Spider)
  def __init__(self):
		custom_settings = {
			'AUTOTHROTTLE_ENABLED': False
		}

  a = scrapy.utils.project.get_project_settings()
  print(a.copy_to_dict())

My settings.py has AUTOTHROTTLE_ENABLED: True.

Based on my reading of https://doc.scrapy.org/en/latest/topics/settings.html#populating-the-settings - the custom_settings value should override the value of settings.py. But the result of the above is:

....
AUTOTHROTTLE_ENABLED: True
...

It's using the value from settings.py - am I misunderstanding this or is it a bug? I was hoping to load and alter settings dynamically during the init.

The two patches

Our patch+106 −1
docs/topics/settings.rstindex 06a03e69e..fb465b529 100644@@ -128,6 +128,17 @@ arguments <spiderargs>` or other logic:                 )             return spider +.. note:: Spider settings are applied to the settings of the+    :class:`~scrapy.crawler.Crawler` before the spider is instantiated, so+    :attr:`~scrapy.Spider.custom_settings` must be defined as a class+    attribute: setting it on the spider instance, e.g. in ``__init__()``, has+    no effect and triggers a warning.++    Spider settings are also not part of the settings returned by+    :func:`scrapy.utils.project.get_project_settings`, which only include+    project settings. To read the settings that apply to a running spider, use+    :attr:`Spider.settings <scrapy.Spider.settings>`.+ .. _project-settings:  3. Project settingsscrapy/crawler.pyindex 304b63d37..2545b8d97 100644@@ -324,7 +324,29 @@ class Crawler:             raise      def _create_spider(self, *args: Any, **kwargs: Any) -> Spider:-        return self.spidercls.from_crawler(self, *args, **kwargs)+        spider = self.spidercls.from_crawler(self, *args, **kwargs)+        self._warn_instance_custom_settings(spider)+        return spider++    def _warn_instance_custom_settings(self, spider: Spider) -> None:+        """Warn if *spider* defines :attr:`~scrapy.Spider.custom_settings` on+        the instance (e.g. in ``__init__()`` or as a spider argument), since+        spider settings are read from the spider class before the spider is+        instantiated, and hence such a value is ignored."""+        if "custom_settings" not in vars(spider):+            return+        if spider.custom_settings is self.spidercls.custom_settings:+            return+        warnings.warn(+            f"{self.spidercls.__qualname__}.custom_settings was set on the "+            f"spider instance (e.g. in __init__() or as a spider argument), "+            f"so it is ignored: spider settings are applied before the "+            f"spider is instantiated. Define custom_settings as a class "+            f"attribute, implement the update_settings() class method, or "+            f"set settings with priority='spider' in from_crawler() instead.",+            category=UserWarning,+            stacklevel=3,+        )      def _create_engine(self) -> ExecutionEngine:         return ExecutionEngine(self, lambda _: self.stop_async())scrapy/utils/project.pyindex 3e75c7729..46e063830 100644@@ -64,6 +64,16 @@ def data_path(path: str | os.PathLike[str], createdir: bool = False) -> str:   def get_project_settings() -> Settings:+    """Return a :class:`~scrapy.settings.Settings` object with the+    :ref:`project settings <project-settings>` of the current project, on top+    of the :ref:`global default settings <default-settings>`.++    The returned settings do not include :ref:`spider settings+    <spider-settings>` or :ref:`add-on settings <addon-settings>`, which are+    only applied to the settings of a :class:`~scrapy.crawler.Crawler` when it+    is created for a given spider class. To read the settings that apply to a+    running spider, use :attr:`Spider.settings <scrapy.Spider.settings>`.+    """     if ENVVAR not in os.environ:         project = os.environ.get("SCRAPY_PROJECT", "default")         init_env(project)tests/test_crawler.pyindex 9e22839a1..156780aaa 100644@@ -539,6 +539,68 @@ class TestSpiderSettings:         enabled_exts = [e.__class__ for e in crawler.extensions.middlewares]         assert AutoThrottle in enabled_exts +    def test_spider_custom_settings_class_attribute_no_warning(+        self, recwarn: pytest.WarningsRecorder+    ) -> None:+        class MySpider(scrapy.Spider):+            name = "spider"+            custom_settings = {"AUTOTHROTTLE_ENABLED": False}++        crawler = get_raw_crawler(MySpider, BASE_SETTINGS)+        crawler._create_spider()+        assert not [w for w in recwarn if "custom_settings" in str(w.message)]+        assert crawler.settings.getbool("AUTOTHROTTLE_ENABLED") is False++    def test_spider_custom_settings_set_in_init_warns(self) -> None:+        class MySpider(scrapy.Spider):+            name = "spider"++            def __init__(self, *args: Any, **kwargs: Any):+                super().__init__(*args, **kwargs)+                self.custom_settings = {"AUTOTHROTTLE_ENABLED": True}++        crawler = get_raw_crawler(MySpider, BASE_SETTINGS)+        with pytest.warns(+            UserWarning, match=r"MySpider\.custom_settings was set on the spider"+        ):+            crawler._create_spider()+        assert crawler.settings.getbool("AUTOTHROTTLE_ENABLED") is False++    def test_spider_custom_settings_spider_argument_warns(self) -> None:+        class MySpider(scrapy.Spider):+            name = "spider"++        crawler = get_raw_crawler(MySpider, BASE_SETTINGS)+        with pytest.warns(UserWarning, match=r"custom_settings was set on the spider"):+            crawler._create_spider(custom_settings="{'AUTOTHROTTLE_ENABLED': True}")+        assert crawler.settings.getbool("AUTOTHROTTLE_ENABLED") is False++    @coroutine_test+    async def test_spider_custom_settings_set_in_init_warns_crawl(+        self, recwarn: pytest.WarningsRecorder+    ) -> None:+        class MySpider(NoRequestsSpider):+            custom_settings = {"AUTOTHROTTLE_ENABLED": False}++            def __init__(self, *args: Any, **kwargs: Any):+                super().__init__(*args, **kwargs)+                self.custom_settings = {"AUTOTHROTTLE_ENABLED": True}++        crawler = get_raw_crawler(MySpider, BASE_SETTINGS)+        # recwarn instead of pytest.warns(), which would re-emit unrelated+        # warnings (e.g. a ResourceWarning at an arbitrary GC point during the+        # crawl) and fail the test.+        await crawler.crawl_async()+        assert [+            w+            for w in recwarn+            if issubclass(w.category, UserWarning)+            and "MySpider.custom_settings was set on the spider" in str(w.message)+        ]+        assert crawler.settings.getbool("AUTOTHROTTLE_ENABLED") is False+        enabled_exts = [e.__class__ for e in crawler.extensions.middlewares]+        assert AutoThrottle not in enabled_exts+  class TestCrawlerLogging:     def test_no_root_handler_installed(self) -> None: 
Merged fix · PR #8051+12 −0
docs/topics/api.rstindex 24f46f656..d6e8a54f9 100644@@ -166,6 +166,8 @@ Settings API  .. autofunction:: get_settings_priority +.. autofunction:: scrapy.utils.project.get_project_settings+ .. autoclass:: Settings    :show-inheritance:    :members:docs/topics/settings.rstindex 06a03e69e..90d2eca29 100644@@ -137,6 +137,10 @@ Scrapy projects include a settings module, usually a file called ``settings.py``, where you should populate most settings that apply to all your spiders. +:func:`scrapy.utils.project.get_project_settings` returns these settings, e.g.+to pass them to :class:`~scrapy.crawler.AsyncCrawlerProcess` when :ref:`running+Scrapy from a script <run-from-script>`.+ .. seealso:: :ref:`topics-settings-module-envvar`  .. _addon-settings:scrapy/utils/project.pyindex 3e75c7729..831ac7770 100644@@ -64,6 +64,12 @@ def data_path(path: str | os.PathLike[str], createdir: bool = False) -> str:   def get_project_settings() -> Settings:+    """Return a :class:`~scrapy.settings.Settings` object with the+    :ref:`project settings <project-settings>` of the current project.++    Settings from sources with a higher precedence, such as :ref:`spider+    settings <spider-settings>`, are applied when a crawl starts.+    """     if ENVVAR not in os.environ:         project = os.environ.get("SCRAPY_PROJECT", "default")         init_env(project)