Backtestsscrapyrun-20261025-211112#6136
Investigate simplifying static checks for initialized Crawler
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches reach the issue's goal: the late-initialized Crawler attributes become non-Optional for mypy and every `assert crawler.stats`-style narrowing is removed, and the independent static-type tests pass on both. They differ in how they get there. Patch A replaces the attributes with a `_LateAttribute` descriptor that raises RuntimeError on read before the crawl starts. That is honest typing with runtime enforcement, but it is a public behavior change with real risk beyond the issue's ask: `crawler.engine`/`crawler.stats` went from returning None to raising, so any third-party `from_crawler` or pre-crawl code that reads them (even just to check `is None`) now crashes; worse, because the error is RuntimeError rather than AttributeError, `hasattr(crawler, 'stats')` and `getattr(crawler, 'stats', None)` also raise, which is a surprising trap. It also removes the attributes from the instance dict and adds ~50 lines of descriptor/overload machinery. A's new tests only exercise the new RuntimeError behavior and class-level access; none check the static types the issue is actually about. Patch B keeps runtime behavior identical (attributes stay None until set) and just annotates them as non-Optional with `# type: ignore[assignment]`, documenting the arrangement in the class comment and the API docs. It is a type-level simplification exactly matching the issue's 'investigate simplifying static checks' framing, with zero runtime risk to Scrapy or third-party code. It also extends the treatment to `spider`, consistent with the issue's 'components such as'. Its runtime test is largely tautological against base, but its new `tests_typing/test_crawler.mypy-testing` file with `reveal_type` checks and an assert-free component sample is a direct regression test for the reported problem that would fail on base and pass after. A's downside is the type lie it accepts, but the project already relies on documented lifecycle guarantees (and RuntimeError-raising getters like get_extension) for these attributes, so the lie is bounded and documented. Weighing correctness/risk, the more on-target tests, and scope, B is the better fix; A's broader redesign adds API breakage without being required by the issue.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
17m 47s
Wall-clock time in the container
Full test suite
No new failures
1,224 passed · 282 skipped · 1,514 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
172vs 231 merged
Changed lines, ours vs merged
The issue
View on GitHubAfter we changed the Crawler logic to initialize components such as Crawler.stats later, static checkers (specifically mypy) don't know that when code like self.crawler.stats.inc_value() is called, self.crawler.stats is already initialized, so we need to add asserts in all methods that do this. We should try to find a way to do the same with fewer asserts. This also applies to 3rd-party code (middlewares etc.).
The two patches
docs/topics/api.rstindex 7ff8d3464..5574b00f2 100644@@ -35,6 +35,17 @@ how you :ref:`configure the downloader middlewares :class:`scrapy.Spider` subclass and a :class:`scrapy.settings.Settings` object. + The :attr:`engine`, :attr:`extensions`, :attr:`logformatter`,+ :attr:`request_fingerprinter`, :attr:`spider` and :attr:`stats` attributes+ are ``None`` until they are set when the crawl starts. When starting a+ crawl, the :attr:`spider` attribute is set first, then the other+ attributes; :attr:`extensions` and+ :attr:`engine` are only set after the components that they manage have been+ created.+ For static type checkers, these attributes are typed as non-optional, so+ code that uses them once they are set doesn't need to narrow their types,+ e.g. with ``assert crawler.stats is not None``.+ .. attribute:: request_fingerprinter The request fingerprint builder of this crawler.scrapy/commands/parse.pyindex 51caed57f..b6255ec7e 100644@@ -281,7 +281,6 @@ class Command(BaseRunSpiderCommand): ) -> list[Any]: items, requests, opts, depth, spider, callback = args if opts.pipelines:- assert self.pcrawler.engine itemproc = self.pcrawler.engine.scraper.itemproc if hasattr(itemproc, "process_item_async"): for item in items:scrapy/core/downloader/handlers/__init__.pyindex 84dc6216b..f64b9fb85 100644@@ -157,7 +157,6 @@ class DownloadHandlers: raise NotSupported( f"Unsupported URL scheme '{scheme}': {self._notconfigured[scheme]}" )- assert self._crawler.spider if scheme in self._old_style_handlers: # pragma: no cover return await maybe_deferred_to_future( cast(scrapy/core/downloader/handlers/http2.pyindex 9b3d4fbd4..67a47b0be 100644@@ -55,7 +55,6 @@ class H2DownloadHandler(BaseDownloadHandler): bind_address=self._bind_address, crawler=self._crawler, )- assert self._crawler.spider with wrap_twisted_exceptions(): return await maybe_deferred_to_future( agent.download_request(request, self._crawler.spider)scrapy/core/engine.pyindex b7a1e0fcd..9601e346d 100644@@ -112,7 +112,6 @@ class ExecutionEngine: self.crawler: Crawler = crawler self.settings: Settings = crawler.settings self.signals: SignalManager = crawler.signals- assert crawler.logformatter self.logformatter: LogFormatter = crawler.logformatter self._slot: _Slot | None = None self.spider: Spider | None = None@@ -299,7 +298,6 @@ class ExecutionEngine: response=None, spider=self.spider, )- assert self.crawler.stats self.crawler.stats.inc_value("spider_exceptions/count") self.crawler.stats.inc_value( f"spider_exceptions/{type(exception).__name__}"@@ -549,7 +547,6 @@ class ExecutionEngine: return deferred_from_coro(self.open_spider_async(close_if_idle=close_if_idle)) async def open_spider_async(self, *, close_if_idle: bool = True) -> None:- assert self.crawler.spider if self._slot is not None: raise RuntimeError( f"No free spider slot when opening {self.crawler.spider.name!r}"@@ -563,7 +560,6 @@ class ExecutionEngine: if hasattr(scheduler, "open") and (d := scheduler.open(self.crawler.spider)): await maybe_deferred_to_future(d) await self.scraper.open_spider_async()- assert self.crawler.stats if argument_is_required(self.crawler.stats.open_spider, "spider"): warnings.warn( f"The open_spider() method of {global_object_name(type(self.crawler.stats))} requires a spider argument,"@@ -676,7 +672,6 @@ class ExecutionEngine: extra={"spider": spider}, ) - assert self.crawler.stats try: if argument_is_required(self.crawler.stats.close_spider, "spider"): warnings.warn(scrapy/core/scraper.pyindex 58e37ce5e..d595641ed 100644@@ -120,7 +120,6 @@ class Scraper: self.concurrent_items: int = crawler.settings.getint("CONCURRENT_ITEMS") self.crawler: Crawler = crawler self.signals: SignalManager = crawler.signals- assert crawler.logformatter self.logformatter: LogFormatter = crawler.logformatter def _check_deprecated_itemproc_method(self, method: str) -> None:@@ -198,7 +197,6 @@ class Scraper: if self._itemproc_has_async["close_spider"]: await self.itemproc.close_spider_async() else:- assert self.crawler.spider await maybe_deferred_to_future( self.itemproc.close_spider(self.crawler.spider) )@@ -210,7 +208,6 @@ class Scraper: def _check_if_closing(self) -> None: assert self.slot is not None # typing if self.slot.closing and self.slot.is_idle():- assert self.crawler.spider self.slot.closing.callback(self.crawler.spider) @inlineCallbacks@@ -266,7 +263,6 @@ class Scraper: output = await self.call_spider_async(result, request) except Exception as spider_exc: # the errback didn't silence the exception- assert self.crawler.spider if not result.check(IgnoreRequest): logkws = self.logformatter.download_error( result, request, self.crawler.spider@@ -310,7 +306,6 @@ class Scraper: .. versionadded:: 2.13 """ await _defer_sleep_async()- assert self.crawler.spider if isinstance(result, Response): if getattr(result, "request", None) is None: result.request = request@@ -352,10 +347,8 @@ class Scraper: response: Response | Failure, ) -> None: """Handle an exception raised by a spider callback or errback."""- assert self.crawler.spider exc = _failure.value if isinstance(exc, CloseSpider):- assert self.crawler.engine is not None # typing _schedule_coro( self.crawler.engine.close_spider_async(reason=exc.reason or "cancelled") )@@ -374,7 +367,6 @@ class Scraper: response=response, spider=self.crawler.spider, )- assert self.crawler.stats self.crawler.stats.inc_value("spider_exceptions/count") self.crawler.stats.inc_value( f"spider_exceptions/{_failure.value.__class__.__name__}"@@ -456,7 +448,6 @@ class Scraper: Items are sent to the item pipelines, requests are scheduled. """ if isinstance(output, Request):- assert self.crawler.engine is not None # typing self.crawler.engine.crawl(request=output) return if output is not None:@@ -488,7 +479,6 @@ class Scraper: .. versionadded:: 2.14 """ assert self.slot is not None # typing- assert self.crawler.spider is not None # typing self.slot.itemproc_size += 1 try: if self._itemproc_has_async["process_item"]:scrapy/crawler.pyindex e2f726519..0002b268d 100644@@ -83,12 +83,20 @@ class Crawler: self.crawling: bool = False self._started: bool = False - self.extensions: ExtensionManager | None = None- self.stats: StatsCollector | None = None- self.logformatter: LogFormatter | None = None- self.request_fingerprinter: RequestFingerprinterProtocol | None = None- self.spider: Spider | None = None- self.engine: ExecutionEngine | None = None+ # The following attributes are None until they are set when the crawl+ # starts (see crawl_async()). They are typed as non-optional so that+ # code that uses them after that, both in Scrapy and in 3rd-party+ # components, doesn't need to narrow their types (e.g. with asserts)+ # for static type checkers. Note that crawl_async() creates the spider+ # before setting the other attributes, and that extensions and engine+ # are set only after all components that they manage have been+ # created.+ self.extensions: ExtensionManager = None # type: ignore[assignment]+ self.stats: StatsCollector = None # type: ignore[assignment]+ self.logformatter: LogFormatter = None # type: ignore[assignment]+ self.request_fingerprinter: RequestFingerprinterProtocol = None # type: ignore[assignment]+ self.spider: Spider = None # type: ignore[assignment]+ self.engine: ExecutionEngine = None # type: ignore[assignment] def _update_root_log_handler(self) -> None: if get_scrapy_root_handler() is not None:scrapy/downloadermiddlewares/httpcache.pyindex e7ca0ac0e..3ebd98027 100644@@ -55,7 +55,6 @@ class HttpCacheMiddleware: @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.stats o = cls(crawler.settings, crawler.stats) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)scrapy/downloadermiddlewares/offsite.pyindex 0d5c9e4c9..eaf98d566 100644@@ -61,7 +61,6 @@ class OffsiteMiddleware: @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.stats o = cls(crawler.stats) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.request_scheduled, signal=signals.request_scheduled)@@ -82,7 +81,6 @@ class OffsiteMiddleware: @_warn_spider_arg def process_request(self, request: Request, spider: Spider | None = None) -> None:- assert self.crawler.spider if ( request.dont_filter or request.meta.get("allow_offsite")scrapy/downloadermiddlewares/retry.pyindex f910d07c8..3ec17ae6e 100644@@ -94,7 +94,6 @@ def get_retry_request( retry-related job stats """ settings = spider.crawler.settings- assert spider.crawler.stats stats = spider.crawler.stats retry_times = request.meta.get("retry_times", 0) + 1 if max_retry_times is None:@@ -192,7 +191,6 @@ class RetryMiddleware: give_up_log_level = request.meta.get( "give_up_log_level", self.give_up_log_level )- assert self.crawler.spider return get_retry_request( request, reason=reason,scrapy/downloadermiddlewares/robotstxt.pyindex 81a3a887f..cca1cc7af 100644@@ -78,7 +78,6 @@ class RobotsTxtMiddleware: {"request": request}, extra={"spider": self.crawler.spider}, )- assert self.crawler.stats self.crawler.stats.inc_value("robotstxt/forbidden") raise IgnoreRequest("Forbidden by robots.txt") @@ -95,8 +94,6 @@ class RobotsTxtMiddleware: meta={"dont_obey_robotstxt": True}, callback=NO_CALLBACK, )- assert self.crawler.engine- assert self.crawler.stats try: resp = await self.crawler.engine.download_async(robotsreq) await self._parse_robots(resp, netloc, request)@@ -119,7 +116,6 @@ class RobotsTxtMiddleware: async def _parse_robots( self, response: Response, netloc: str, request: Request ) -> None:- assert self.crawler.stats self.crawler.stats.inc_value("robotstxt/response_count") self.crawler.stats.inc_value( f"robotstxt/response_status_count/{response.status}"@@ -138,7 +134,6 @@ class RobotsTxtMiddleware: def _robots_error(self, exc: Exception, netloc: str) -> None: if not isinstance(exc, IgnoreRequest): key = f"robotstxt/exception_count/{type(exc)}"- assert self.crawler.stats self.crawler.stats.inc_value(key) rp_dfd = self._parsers[netloc] assert isinstance(rp_dfd, Deferred)scrapy/downloadermiddlewares/stats.pyindex bafa931de..07de1c2e7 100644@@ -43,7 +43,6 @@ class DownloaderStats: def from_crawler(cls, crawler: Crawler) -> Self: if not crawler.settings.getbool("DOWNLOADER_STATS"): raise NotConfigured- assert crawler.stats return cls(crawler.stats) @_warn_spider_argscrapy/dupefilters.pyindex 36fb0f97d..09f0be63b 100644@@ -95,7 +95,6 @@ class RFPDupeFilter(BaseDupeFilter): @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.request_fingerprinter debug = crawler.settings.getbool("DUPEFILTER_DEBUG") return cls( job_dir(crawler.settings),@@ -134,5 +133,4 @@ class RFPDupeFilter(BaseDupeFilter): self.logger.debug(msg, {"request": request}, extra={"spider": spider}) self.logdupes = False - assert spider.crawler.stats spider.crawler.stats.inc_value("dupefilter/filtered")scrapy/extensions/closespider.pyindex 9cb792e30..5d40e5f9f 100644@@ -102,7 +102,6 @@ class CloseSpider: self._close_spider("closespider_pagecount_no_item") def spider_opened(self, spider: Spider) -> None:- assert self.crawler.engine self.task = call_later( self.close_on["timeout"], self._close_spider, "closespider_timeout" )@@ -146,5 +145,4 @@ class CloseSpider: self._close_spider("closespider_timeout_no_item") def _close_spider(self, reason: str) -> None:- assert self.crawler.engine _schedule_coro(self.crawler.engine.close_spider_async(reason=reason))scrapy/extensions/corestats.pyindex 6a5e55992..b464942af 100644@@ -26,7 +26,6 @@ class CoreStats: @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.stats o = cls(crawler.stats) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)scrapy/extensions/debug.pyindex 5def7509e..8802a3b58 100644@@ -45,7 +45,6 @@ class StackTraceDump: return cls(crawler) def dump_stacktrace(self, signum: int, frame: FrameType | None) -> None:- assert self.crawler.engine log_args = { "stackdumps": self._thread_stacks(), "enginestatus": format_engine_status(self.crawler.engine),scrapy/extensions/feedexport.pyindex e4294ac90..32abc82d3 100644@@ -611,7 +611,6 @@ class FeedExporter: logmsg = f"{slot.format} feed ({slot.itemcount} items) in: {slot.uri}" slot_type = type(slot.storage).__name__- assert self.crawler.stats try: await ensure_awaitable(slot.storage.store(self._get_file(slot))) except Exception:scrapy/extensions/httpcache.pyindex dbb79b02d..d008d0f67 100644@@ -261,7 +261,6 @@ class DbmCacheStorage: extra={"spider": spider}, ) - assert spider.crawler.request_fingerprinter self._fingerprinter: RequestFingerprinterProtocol = ( spider.crawler.request_fingerprinter )@@ -326,7 +325,6 @@ class FilesystemCacheStorage: extra={"spider": spider}, ) - assert spider.crawler.request_fingerprinter self._fingerprinter = spider.crawler.request_fingerprinter def close_spider(self, spider: Spider) -> None:scrapy/extensions/logstats.pyindex 6c94d947e..b818569ba 100644@@ -37,7 +37,6 @@ class LogStats: interval: float = crawler.settings.getfloat("LOGSTATS_INTERVAL") if not interval: raise NotConfigured- assert crawler.stats o = cls(crawler.stats, interval) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)scrapy/extensions/memdebug.pyindex 1fde6b296..35ff90e3b 100644@@ -29,7 +29,6 @@ class MemoryDebugger: def from_crawler(cls, crawler: Crawler) -> Self: if not crawler.settings.getbool("MEMDEBUG_ENABLED"): raise NotConfigured- assert crawler.stats o = cls(crawler.stats) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed) return oscrapy/extensions/memusage.pyindex e0e289ce8..842d29182 100644@@ -77,7 +77,6 @@ class MemoryUsage: return size def engine_started(self) -> None:- assert self.crawler.stats self.crawler.stats.set_value("memusage/startup", self.get_virtual_size()) self.tasks: list[AsyncioLoopingCall | LoopingCall] = [] tsk = create_looping_call(self.update)@@ -98,12 +97,9 @@ class MemoryUsage: tsk.stop() def update(self) -> None:- assert self.crawler.stats self.crawler.stats.max_value("memusage/max", self.get_virtual_size()) def _check_limit(self) -> None:- assert self.crawler.engine- assert self.crawler.stats peak_mem_usage = self.get_virtual_size() if peak_mem_usage > self.limit: self.crawler.stats.set_value("memusage/limit_reached", 1)@@ -136,7 +132,6 @@ class MemoryUsage: def _check_warning(self) -> None: if self.warned: # warn only once return- assert self.crawler.stats if self.get_virtual_size() > self.warning: self.crawler.stats.set_value("memusage/warning_reached", 1) self.crawler.signals.send_catch_log(signal=signals.memusage_warning_reached)@@ -157,8 +152,6 @@ class MemoryUsage: def _send_report(self, rcpts: list[str], subject: str) -> None: # pragma: no cover """send notification mail with some additional useful info"""- assert self.crawler.engine- assert self.crawler.stats stats = self.crawler.stats s = f"Memory usage at engine startup : {stats.get_value('memusage/startup') / 1024 / 1024}M\r\n" s += f"Maximum memory usage : {stats.get_value('memusage/max') / 1024 / 1024}M\r\n"scrapy/extensions/periodic_log.pyindex cbcc8b70e..adffbcbc4 100644@@ -87,7 +87,6 @@ class PeriodicLog: ) if not (ext_stats or ext_delta or ext_timing_enabled): raise NotConfigured- assert crawler.stats assert ext_stats is not None assert ext_delta is not None o = cls(scrapy/extensions/statsmailer.pyindex f05595806..7647cf33d 100644@@ -42,7 +42,6 @@ class StatsMailer: if not recipients: raise NotConfigured mail: MailSender = MailSender.from_crawler(crawler)- assert crawler.stats o = cls(crawler.stats, recipients, mail) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed) return oscrapy/extensions/telnet.pyindex 1506cb1ea..392f79299 100644@@ -108,7 +108,6 @@ class TelnetConsole(protocol.ServerFactory): def _get_telnet_vars(self) -> dict[str, Any]: # Note: if you add entries here also update topics/telnetconsole.rst- assert self.crawler.engine telnet_vars: dict[str, Any] = { "engine": self.crawler.engine, "spider": self.crawler.engine.spider,scrapy/extensions/throttle.pyindex cde73f12e..f44aee03d 100644@@ -45,7 +45,6 @@ class AutoThrottle: def _spider_opened(self, spider: Spider) -> None: self.mindelay = self._min_delay() self.maxdelay = self._max_delay()- assert self.crawler.engine self.crawler.engine.downloader._delay = self._start_delay() def _min_delay(self) -> float:@@ -98,7 +97,6 @@ class AutoThrottle: key: str | None = request.meta.get("download_slot") if key is None: return None, None- assert self.crawler.engine return key, self.crawler.engine.downloader.slots.get(key) def _adjust_delay(self, slot: Slot, latency: float, response: Response) -> None:scrapy/pipelines/files.pyindex 55a3676e5..69e347965 100644@@ -697,7 +697,6 @@ class FilesPipeline(MediaPipeline): } def inc_stats(self, status: str) -> None:- assert self.crawler.stats self.crawler.stats.inc_value("file_count") self.crawler.stats.inc_value(f"file_status_count/{status}") scrapy/pipelines/media.pyindex 764f82a78..5d5985617 100644@@ -100,7 +100,6 @@ class MediaPipeline(ABC): stacklevel=2, ) self.crawler: Crawler = crawler- assert crawler.request_fingerprinter self._fingerprinter: RequestFingerprinterProtocol = ( crawler.request_fingerprinter )@@ -141,7 +140,6 @@ class MediaPipeline(ABC): @_warn_spider_arg def open_spider(self, spider: Spider | None = None) -> None:- assert self.crawler.spider self.spiderinfo = self.SpiderInfo(self.crawler.spider) @_warn_spider_arg@@ -228,7 +226,6 @@ class MediaPipeline(ABC): ) -> FileInfo: try: self._modify_media_request(request)- assert self.crawler.engine response = await self.crawler.engine.download_async(request) return await ensure_awaitable( self.media_downloaded(response, request, info, item=item)scrapy/pqueues.pyindex 8e9783f5a..f4c35c0fe 100644@@ -265,7 +265,6 @@ class ScrapyPriorityQueue: class DownloaderInterface: def __init__(self, crawler: Crawler):- assert crawler.engine self.downloader: Downloader = crawler.engine.downloader def stats(self, possible_slots: Iterable[str]) -> list[tuple[int, str]]:scrapy/shell.pyindex dfea00c46..8f78af571 100644@@ -193,7 +193,6 @@ class Shell: """ if not self.spider: await self._open_spider(spider)- assert self.crawler.engine is not None # send the request to the engine self.crawler.engine.crawl(request) # this will fire when the request callback runs (via the callback hijacking in _request_deferred())@@ -204,7 +203,6 @@ class Shell: spider = self.crawler.spider or self.crawler._create_spider() self.crawler.spider = spider- assert self.crawler.engine await self.crawler.engine.open_spider_async(close_if_idle=False) self.spider = spider scrapy/spidermiddlewares/depth.pyindex 1781543e3..49683168e 100644@@ -70,7 +70,6 @@ class DepthMiddleware(BaseSpiderMiddleware): maxdepth = settings.getint("DEPTH_LIMIT") verbose = settings.getbool("DEPTH_STATS_VERBOSE") prio = settings.getint("DEPTH_PRIORITY")- assert crawler.stats o = cls(maxdepth, crawler.stats, verbose, prio) o.crawler = crawler return oscrapy/spidermiddlewares/httperror.pyindex 156b73e7e..9551be371 100644@@ -78,7 +78,6 @@ class HttpErrorMiddleware: self, response: Response, exception: Exception, spider: Spider | None = None ) -> Iterable[Any] | None: if isinstance(exception, HttpError):- assert self.crawler.stats self.crawler.stats.inc_value("httperror/response_ignored_count") self.crawler.stats.inc_value( f"httperror/response_ignored_status_count/{response.status}"scrapy/spidermiddlewares/urllength.pyindex f325ce7a0..86bdd2ed6 100644@@ -48,6 +48,5 @@ class UrlLengthMiddleware(BaseSpiderMiddleware): {"maxlength": self.maxlength, "url": request.url}, extra={"spider": self.crawler.spider}, )- assert self.crawler.stats self.crawler.stats.inc_value("urllength/request_ignored_count") return Nonescrapy/utils/log.pyindex ee357c031..bfa39169f 100644@@ -239,7 +239,6 @@ class LogCounterHandler(logging.Handler): def emit(self, record: logging.LogRecord) -> None: sname = f"log_count/{record.levelname}"- assert self.crawler.stats self.crawler.stats.inc_value(sname) tests/test_crawler.pyindex 358f20ed7..188141250 100644@@ -153,6 +153,25 @@ class TestCrawler: with pytest.raises(RuntimeError, match="Crawling already taking place"): await t2 + @coroutine_test+ async def test_crawler_attributes_initialization(self) -> None:+ attrs = (+ "extensions",+ "stats",+ "logformatter",+ "request_fingerprinter",+ "spider",+ "engine",+ )+ crawler = get_raw_crawler(NoRequestsSpider, BASE_SETTINGS)+ for attr in attrs:+ assert getattr(crawler, attr) is None+ await crawler.crawl_async()+ for attr in attrs:+ assert getattr(crawler, attr) is not None+ assert isinstance(crawler.spider, NoRequestsSpider)+ assert crawler.stats.get_value("finish_reason") == "finished"+ def test_get_addon(self) -> None: class ParentAddon: passtests_typing/test_crawler.mypy-testingnew file mode 100644index 000000000..0ba3307c7@@ -0,0 +1,59 @@+import pytest++from scrapy.crawler import Crawler+from scrapy.spiders import Spider+++@pytest.mark.mypy_testing+def test_crawler_extensions() -> None:+ crawler = Crawler(Spider)+ reveal_type(crawler.extensions) # R: scrapy.extension.ExtensionManager+++@pytest.mark.mypy_testing+def test_crawler_stats() -> None:+ crawler = Crawler(Spider)+ reveal_type(crawler.stats) # R: scrapy.statscollectors.StatsCollector+++@pytest.mark.mypy_testing+def test_crawler_logformatter() -> None:+ crawler = Crawler(Spider)+ reveal_type(crawler.logformatter) # R: scrapy.logformatter.LogFormatter+++@pytest.mark.mypy_testing+def test_crawler_request_fingerprinter() -> None:+ crawler = Crawler(Spider)+ reveal_type(crawler.request_fingerprinter) # R: scrapy.utils.request.RequestFingerprinterProtocol+++@pytest.mark.mypy_testing+def test_crawler_spider() -> None:+ crawler = Crawler(Spider)+ reveal_type(crawler.spider) # R: scrapy.spiders.Spider+++@pytest.mark.mypy_testing+def test_crawler_engine() -> None:+ crawler = Crawler(Spider)+ reveal_type(crawler.engine) # R: scrapy.core.engine.ExecutionEngine+++class StatsComponent:+ def __init__(self, crawler: Crawler):+ self.crawler = crawler++ def on_event(self) -> None:+ self.crawler.stats.inc_value("foo")+ self.crawler.engine.pause()+ self.crawler.spider.logger.info("bar")+++@pytest.mark.mypy_testing+def test_crawler_attributes_no_asserts() -> None:+ crawler = Crawler(Spider)+ crawler.stats.inc_value("foo")+ crawler.engine.unpause()+ crawler.request_fingerprinter.fingerprint+ StatsComponent(crawler).on_event()
docs/topics/api.rstindex 7ff8d3464..598edfeb5 100644@@ -35,6 +35,13 @@ how you :ref:`configure the downloader middlewares :class:`scrapy.Spider` subclass and a :class:`scrapy.settings.Settings` object. + The :attr:`engine`, :attr:`extensions`, :attr:`logformatter`,+ :attr:`request_fingerprinter` and :attr:`stats` attributes get their value+ when the crawl starts, and raise :exc:`RuntimeError` when read before that.++ .. versionchanged:: VERSION+ Those attributes used to be ``None`` before getting their value.+ .. attribute:: request_fingerprinter The request fingerprint builder of this crawler.scrapy/commands/parse.pyindex 51caed57f..b6255ec7e 100644@@ -281,7 +281,6 @@ class Command(BaseRunSpiderCommand): ) -> list[Any]: items, requests, opts, depth, spider, callback = args if opts.pipelines:- assert self.pcrawler.engine itemproc = self.pcrawler.engine.scraper.itemproc if hasattr(itemproc, "process_item_async"): for item in items:scrapy/core/engine.pyindex b7a1e0fcd..b63d136a4 100644@@ -112,7 +112,6 @@ class ExecutionEngine: self.crawler: Crawler = crawler self.settings: Settings = crawler.settings self.signals: SignalManager = crawler.signals- assert crawler.logformatter self.logformatter: LogFormatter = crawler.logformatter self._slot: _Slot | None = None self.spider: Spider | None = None@@ -299,7 +298,6 @@ class ExecutionEngine: response=None, spider=self.spider, )- assert self.crawler.stats self.crawler.stats.inc_value("spider_exceptions/count") self.crawler.stats.inc_value( f"spider_exceptions/{type(exception).__name__}"@@ -563,17 +561,17 @@ class ExecutionEngine: if hasattr(scheduler, "open") and (d := scheduler.open(self.crawler.spider)): await maybe_deferred_to_future(d) await self.scraper.open_spider_async()- assert self.crawler.stats- if argument_is_required(self.crawler.stats.open_spider, "spider"):+ stats = self.crawler.stats+ if argument_is_required(stats.open_spider, "spider"): warnings.warn(- f"The open_spider() method of {global_object_name(type(self.crawler.stats))} requires a spider argument,"+ f"The open_spider() method of {global_object_name(type(stats))} requires a spider argument," f" this is deprecated and the argument will not be passed in future Scrapy versions.", ScrapyDeprecationWarning, stacklevel=2, )- self.crawler.stats.open_spider(spider=self.crawler.spider)+ stats.open_spider(spider=self.crawler.spider) else:- self.crawler.stats.open_spider()+ stats.open_spider() await self.signals.send_catch_log_async( signals.spider_opened, spider=self.crawler.spider )@@ -676,20 +674,18 @@ class ExecutionEngine: extra={"spider": spider}, ) - assert self.crawler.stats try:- if argument_is_required(self.crawler.stats.close_spider, "spider"):+ stats = self.crawler.stats+ if argument_is_required(stats.close_spider, "spider"): warnings.warn(- f"The close_spider() method of {global_object_name(type(self.crawler.stats))} requires a spider argument,"+ f"The close_spider() method of {global_object_name(type(stats))} requires a spider argument," f" this is deprecated and the argument will not be passed in future Scrapy versions.", ScrapyDeprecationWarning, stacklevel=2, )- self.crawler.stats.close_spider(- spider=self.crawler.spider, reason=reason- )+ stats.close_spider(spider=self.crawler.spider, reason=reason) else:- self.crawler.stats.close_spider(reason=reason)+ stats.close_spider(reason=reason) except Exception: logger.error("Stats close failure") scrapy/core/scraper.pyindex 58e37ce5e..6426b1751 100644@@ -120,7 +120,6 @@ class Scraper: self.concurrent_items: int = crawler.settings.getint("CONCURRENT_ITEMS") self.crawler: Crawler = crawler self.signals: SignalManager = crawler.signals- assert crawler.logformatter self.logformatter: LogFormatter = crawler.logformatter def _check_deprecated_itemproc_method(self, method: str) -> None:@@ -355,7 +354,6 @@ class Scraper: assert self.crawler.spider exc = _failure.value if isinstance(exc, CloseSpider):- assert self.crawler.engine is not None # typing _schedule_coro( self.crawler.engine.close_spider_async(reason=exc.reason or "cancelled") )@@ -374,11 +372,9 @@ class Scraper: response=response, spider=self.crawler.spider, )- assert self.crawler.stats- self.crawler.stats.inc_value("spider_exceptions/count")- self.crawler.stats.inc_value(- f"spider_exceptions/{_failure.value.__class__.__name__}"- )+ stats = self.crawler.stats+ stats.inc_value("spider_exceptions/count")+ stats.inc_value(f"spider_exceptions/{_failure.value.__class__.__name__}") def handle_spider_output( self,@@ -456,7 +452,6 @@ class Scraper: Items are sent to the item pipelines, requests are scheduled. """ if isinstance(output, Request):- assert self.crawler.engine is not None # typing self.crawler.engine.crawl(request=output) return if output is not None:scrapy/crawler.pyindex e2f726519..44c4ffdcf 100644@@ -8,7 +8,7 @@ import signal import warnings from abc import ABC, abstractmethod from functools import partial-from typing import TYPE_CHECKING, Any, TypeVar+from typing import TYPE_CHECKING, Any, Generic, TypeVar, overload from twisted.internet.defer import Deferred, DeferredList, inlineCallbacks @@ -58,7 +58,55 @@ logger = logging.getLogger(__name__) _T = TypeVar("_T") +class _LateAttribute(Generic[_T]):+ """Descriptor for a :class:`Crawler` attribute that only gets a value once+ the crawl starts.++ The value is kept in an attribute of the same name prefixed with an+ underscore, and reading it before it is set raises :exc:`RuntimeError`.+ This way the public attribute can be annotated as always set, and its+ users, both in Scrapy and in third-party code, do not need to narrow its+ type on every use. Code that runs before the crawl starts reads the+ underscore-prefixed attribute instead.+ """++ def __set_name__(self, owner: type[Crawler], name: str) -> None:+ self._name = name+ self._private_name = f"_{name}"++ @overload+ def __get__(self, instance: None, owner: type[Crawler]) -> _LateAttribute[_T]: ...++ @overload+ def __get__(self, instance: Crawler, owner: type[Crawler]) -> _T: ...++ def __get__(+ self, instance: Crawler | None, owner: type[Crawler]+ ) -> _LateAttribute[_T] | _T:+ if instance is None:+ return self+ value: _T | None = getattr(instance, self._private_name)+ if value is None:+ raise RuntimeError(+ f"Crawler.{self._name} is not set yet. It is set when the "+ "crawl starts, so it can only be used from then on, e.g. "+ "from the spider_opened signal handler onwards."+ )+ return value++ def __set__(self, instance: Crawler, value: _T) -> None:+ setattr(instance, self._private_name, value)++ class Crawler:+ engine: _LateAttribute[ExecutionEngine] = _LateAttribute()+ extensions: _LateAttribute[ExtensionManager] = _LateAttribute()+ logformatter: _LateAttribute[LogFormatter] = _LateAttribute()+ request_fingerprinter: _LateAttribute[RequestFingerprinterProtocol] = (+ _LateAttribute()+ )+ stats: _LateAttribute[StatsCollector] = _LateAttribute()+ def __init__( self, spidercls: type[Spider],@@ -83,12 +131,13 @@ class Crawler: self.crawling: bool = False self._started: bool = False - self.extensions: ExtensionManager | None = None- self.stats: StatsCollector | None = None- self.logformatter: LogFormatter | None = None- self.request_fingerprinter: RequestFingerprinterProtocol | None = None self.spider: Spider | None = None- self.engine: ExecutionEngine | None = None++ self._engine: ExecutionEngine | None = None+ self._extensions: ExtensionManager | None = None+ self._logformatter: LogFormatter | None = None+ self._request_fingerprinter: RequestFingerprinterProtocol | None = None+ self._stats: StatsCollector | None = None def _update_root_log_handler(self) -> None: if get_scrapy_root_handler() is not None:@@ -225,8 +274,8 @@ class Crawler: yield deferred_from_coro(self.engine.start_async()) except Exception: self.crawling = False- if self.engine is not None:- yield deferred_from_coro(self.engine.close_async())+ if self._engine is not None:+ yield deferred_from_coro(self._engine.close_async()) raise async def crawl_async(self, *args: Any, **kwargs: Any) -> None:@@ -255,8 +304,8 @@ class Crawler: await self.engine.start_async() except Exception: self.crawling = False- if self.engine is not None:- await self.engine.close_async()+ if self._engine is not None:+ await self._engine.close_async() raise def _create_spider(self, *args: Any, **kwargs: Any) -> Spider:@@ -282,7 +331,6 @@ class Crawler: """ if self.crawling: self.crawling = False- assert self.engine if self.engine.running: await self.engine.stop_async() @@ -313,7 +361,7 @@ class Crawler: This method can only be called after the crawl engine has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. """- if not self.engine:+ if self._engine is None: raise RuntimeError( "Crawler.get_downloader_middleware() can only be called after " "the crawl engine has been created."@@ -331,7 +379,7 @@ class Crawler: created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. """- if not self.extensions:+ if self._extensions is None: raise RuntimeError( "Crawler.get_extension() can only be called after the " "extension manager has been created."@@ -348,7 +396,7 @@ class Crawler: This method can only be called after the crawl engine has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. """- if not self.engine:+ if self._engine is None: raise RuntimeError( "Crawler.get_item_pipeline() can only be called after the " "crawl engine has been created."@@ -365,7 +413,7 @@ class Crawler: This method can only be called after the crawl engine has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. """- if not self.engine:+ if self._engine is None: raise RuntimeError( "Crawler.get_spider_middleware() can only be called after the " "crawl engine has been created."scrapy/downloadermiddlewares/httpcache.pyindex e7ca0ac0e..3ebd98027 100644@@ -55,7 +55,6 @@ class HttpCacheMiddleware: @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.stats o = cls(crawler.settings, crawler.stats) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)scrapy/downloadermiddlewares/offsite.pyindex 0d5c9e4c9..b9a26df3a 100644@@ -61,7 +61,6 @@ class OffsiteMiddleware: @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.stats o = cls(crawler.stats) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.request_scheduled, signal=signals.request_scheduled)scrapy/downloadermiddlewares/retry.pyindex f910d07c8..dd2897ee4 100644@@ -94,7 +94,6 @@ def get_retry_request( retry-related job stats """ settings = spider.crawler.settings- assert spider.crawler.stats stats = spider.crawler.stats retry_times = request.meta.get("retry_times", 0) + 1 if max_retry_times is None:scrapy/downloadermiddlewares/robotstxt.pyindex 81a3a887f..d7aa8738c 100644@@ -27,6 +27,7 @@ if TYPE_CHECKING: from scrapy import Spider from scrapy.crawler import Crawler from scrapy.robotstxt import RobotParser+ from scrapy.statscollectors import StatsCollector logger = logging.getLogger(__name__)@@ -41,6 +42,7 @@ class RobotsTxtMiddleware: self._default_useragent: str = crawler.settings["USER_AGENT"] self._robotstxt_useragent: str | None = crawler.settings["ROBOTSTXT_USER_AGENT"] self.crawler: Crawler = crawler+ self._stats: StatsCollector = crawler.stats self._parsers: dict[str, RobotParser | Deferred[RobotParser | None] | None] = {} self._parserimpl: RobotParser = load_object( crawler.settings.get("ROBOTSTXT_PARSER")@@ -78,8 +80,7 @@ class RobotsTxtMiddleware: {"request": request}, extra={"spider": self.crawler.spider}, )- assert self.crawler.stats- self.crawler.stats.inc_value("robotstxt/forbidden")+ self._stats.inc_value("robotstxt/forbidden") raise IgnoreRequest("Forbidden by robots.txt") async def robot_parser(self, request: Request) -> RobotParser | None:@@ -95,8 +96,6 @@ class RobotsTxtMiddleware: meta={"dont_obey_robotstxt": True}, callback=NO_CALLBACK, )- assert self.crawler.engine- assert self.crawler.stats try: resp = await self.crawler.engine.download_async(robotsreq) await self._parse_robots(resp, netloc, request)@@ -109,7 +108,7 @@ class RobotsTxtMiddleware: extra={"spider": self.crawler.spider}, ) self._robots_error(e, netloc)- self.crawler.stats.inc_value("robotstxt/request_count")+ self._stats.inc_value("robotstxt/request_count") parser = self._parsers[netloc] if isinstance(parser, Deferred):@@ -119,11 +118,8 @@ class RobotsTxtMiddleware: async def _parse_robots( self, response: Response, netloc: str, request: Request ) -> None:- assert self.crawler.stats- self.crawler.stats.inc_value("robotstxt/response_count")- self.crawler.stats.inc_value(- f"robotstxt/response_status_count/{response.status}"- )+ self._stats.inc_value("robotstxt/response_count")+ self._stats.inc_value(f"robotstxt/response_status_count/{response.status}") rp = self._parserimpl.from_crawler(self.crawler, response.body) await self.crawler.signals.send_catch_log_async( signal=signals.robots_parsed,@@ -138,8 +134,7 @@ class RobotsTxtMiddleware: def _robots_error(self, exc: Exception, netloc: str) -> None: if not isinstance(exc, IgnoreRequest): key = f"robotstxt/exception_count/{type(exc)}"- assert self.crawler.stats- self.crawler.stats.inc_value(key)+ self._stats.inc_value(key) rp_dfd = self._parsers[netloc] assert isinstance(rp_dfd, Deferred) self._parsers[netloc] = Nonescrapy/downloadermiddlewares/stats.pyindex bafa931de..07de1c2e7 100644@@ -43,7 +43,6 @@ class DownloaderStats: def from_crawler(cls, crawler: Crawler) -> Self: if not crawler.settings.getbool("DOWNLOADER_STATS"): raise NotConfigured- assert crawler.stats return cls(crawler.stats) @_warn_spider_argscrapy/dupefilters.pyindex 36fb0f97d..09f0be63b 100644@@ -95,7 +95,6 @@ class RFPDupeFilter(BaseDupeFilter): @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.request_fingerprinter debug = crawler.settings.getbool("DUPEFILTER_DEBUG") return cls( job_dir(crawler.settings),@@ -134,5 +133,4 @@ class RFPDupeFilter(BaseDupeFilter): self.logger.debug(msg, {"request": request}, extra={"spider": spider}) self.logdupes = False - assert spider.crawler.stats spider.crawler.stats.inc_value("dupefilter/filtered")scrapy/extensions/closespider.pyindex 9cb792e30..5d40e5f9f 100644@@ -102,7 +102,6 @@ class CloseSpider: self._close_spider("closespider_pagecount_no_item") def spider_opened(self, spider: Spider) -> None:- assert self.crawler.engine self.task = call_later( self.close_on["timeout"], self._close_spider, "closespider_timeout" )@@ -146,5 +145,4 @@ class CloseSpider: self._close_spider("closespider_timeout_no_item") def _close_spider(self, reason: str) -> None:- assert self.crawler.engine _schedule_coro(self.crawler.engine.close_spider_async(reason=reason))scrapy/extensions/corestats.pyindex 6a5e55992..b464942af 100644@@ -26,7 +26,6 @@ class CoreStats: @classmethod def from_crawler(cls, crawler: Crawler) -> Self:- assert crawler.stats o = cls(crawler.stats) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)scrapy/extensions/debug.pyindex 5def7509e..8802a3b58 100644@@ -45,7 +45,6 @@ class StackTraceDump: return cls(crawler) def dump_stacktrace(self, signum: int, frame: FrameType | None) -> None:- assert self.crawler.engine log_args = { "stackdumps": self._thread_stacks(), "enginestatus": format_engine_status(self.crawler.engine),scrapy/extensions/feedexport.pyindex e4294ac90..32abc82d3 100644@@ -611,7 +611,6 @@ class FeedExporter: logmsg = f"{slot.format} feed ({slot.itemcount} items) in: {slot.uri}" slot_type = type(slot.storage).__name__- assert self.crawler.stats try: await ensure_awaitable(slot.storage.store(self._get_file(slot))) except Exception:scrapy/extensions/httpcache.pyindex dbb79b02d..d008d0f67 100644@@ -261,7 +261,6 @@ class DbmCacheStorage: extra={"spider": spider}, ) - assert spider.crawler.request_fingerprinter self._fingerprinter: RequestFingerprinterProtocol = ( spider.crawler.request_fingerprinter )@@ -326,7 +325,6 @@ class FilesystemCacheStorage: extra={"spider": spider}, ) - assert spider.crawler.request_fingerprinter self._fingerprinter = spider.crawler.request_fingerprinter def close_spider(self, spider: Spider) -> None:scrapy/extensions/logstats.pyindex 6c94d947e..b818569ba 100644@@ -37,7 +37,6 @@ class LogStats: interval: float = crawler.settings.getfloat("LOGSTATS_INTERVAL") if not interval: raise NotConfigured- assert crawler.stats o = cls(crawler.stats, interval) crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)scrapy/extensions/memdebug.pyindex 1fde6b296..35ff90e3b 100644@@ -29,7 +29,6 @@ class MemoryDebugger: def from_crawler(cls, crawler: Crawler) -> Self: if not crawler.settings.getbool("MEMDEBUG_ENABLED"): raise NotConfigured- assert crawler.stats o = cls(crawler.stats) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed) return oscrapy/extensions/memusage.pyindex e0e289ce8..ec761cfbf 100644@@ -27,6 +27,7 @@ if TYPE_CHECKING: from typing_extensions import Self from scrapy.crawler import Crawler+ from scrapy.statscollectors import StatsCollector logger = logging.getLogger(__name__)@@ -43,6 +44,7 @@ class MemoryUsage: raise NotConfigured from exc self.crawler: Crawler = crawler+ self._stats: StatsCollector = crawler.stats self.warned: bool = False self.notify_mails: list[str] = crawler.settings.getlist("MEMUSAGE_NOTIFY_MAIL") if self.notify_mails: # pragma: no cover@@ -77,8 +79,7 @@ class MemoryUsage: return size def engine_started(self) -> None:- assert self.crawler.stats- self.crawler.stats.set_value("memusage/startup", self.get_virtual_size())+ self._stats.set_value("memusage/startup", self.get_virtual_size()) self.tasks: list[AsyncioLoopingCall | LoopingCall] = [] tsk = create_looping_call(self.update) self.tasks.append(tsk)@@ -98,15 +99,12 @@ class MemoryUsage: tsk.stop() def update(self) -> None:- assert self.crawler.stats- self.crawler.stats.max_value("memusage/max", self.get_virtual_size())+ self._stats.max_value("memusage/max", self.get_virtual_size()) def _check_limit(self) -> None:- assert self.crawler.engine- assert self.crawler.stats peak_mem_usage = self.get_virtual_size() if peak_mem_usage > self.limit:- self.crawler.stats.set_value("memusage/limit_reached", 1)+ self._stats.set_value("memusage/limit_reached", 1) mem = self.limit / 1024 / 1024 logger.error( "Memory usage exceeded %(memusage)dMiB. Shutting down Scrapy...",@@ -119,7 +117,7 @@ class MemoryUsage: f"memory usage exceeded {mem}MiB at {socket.gethostname()}" ) self._send_report(self.notify_mails, subj)- self.crawler.stats.set_value("memusage/limit_notified", 1)+ self._stats.set_value("memusage/limit_notified", 1) if self.crawler.engine.spider is not None: _schedule_coro(@@ -136,9 +134,8 @@ class MemoryUsage: def _check_warning(self) -> None: if self.warned: # warn only once return- assert self.crawler.stats if self.get_virtual_size() > self.warning:- self.crawler.stats.set_value("memusage/warning_reached", 1)+ self._stats.set_value("memusage/warning_reached", 1) self.crawler.signals.send_catch_log(signal=signals.memusage_warning_reached) mem = self.warning / 1024 / 1024 logger.warning(@@ -152,16 +149,13 @@ class MemoryUsage: f"memory usage reached {mem}MiB at {socket.gethostname()}" ) self._send_report(self.notify_mails, subj)- self.crawler.stats.set_value("memusage/warning_notified", 1)+ self._stats.set_value("memusage/warning_notified", 1) self.warned = True def _send_report(self, rcpts: list[str], subject: str) -> None: # pragma: no cover """send notification mail with some additional useful info"""- assert self.crawler.engine- assert self.crawler.stats- stats = self.crawler.stats- s = f"Memory usage at engine startup : {stats.get_value('memusage/startup') / 1024 / 1024}M\r\n"- s += f"Maximum memory usage : {stats.get_value('memusage/max') / 1024 / 1024}M\r\n"+ s = f"Memory usage at engine startup : {self._stats.get_value('memusage/startup') / 1024 / 1024}M\r\n"+ s += f"Maximum memory usage : {self._stats.get_value('memusage/max') / 1024 / 1024}M\r\n" s += f"Current memory usage : {self.get_virtual_size() / 1024 / 1024}M\r\n" s += (scrapy/extensions/periodic_log.pyindex cbcc8b70e..adffbcbc4 100644@@ -87,7 +87,6 @@ class PeriodicLog: ) if not (ext_stats or ext_delta or ext_timing_enabled): raise NotConfigured- assert crawler.stats assert ext_stats is not None assert ext_delta is not None o = cls(scrapy/extensions/statsmailer.pyindex f05595806..7647cf33d 100644@@ -42,7 +42,6 @@ class StatsMailer: if not recipients: raise NotConfigured mail: MailSender = MailSender.from_crawler(crawler)- assert crawler.stats o = cls(crawler.stats, recipients, mail) crawler.signals.connect(o.spider_closed, signal=signals.spider_closed) return oscrapy/extensions/telnet.pyindex 1506cb1ea..392f79299 100644@@ -108,7 +108,6 @@ class TelnetConsole(protocol.ServerFactory): def _get_telnet_vars(self) -> dict[str, Any]: # Note: if you add entries here also update topics/telnetconsole.rst- assert self.crawler.engine telnet_vars: dict[str, Any] = { "engine": self.crawler.engine, "spider": self.crawler.engine.spider,scrapy/extensions/throttle.pyindex cde73f12e..f44aee03d 100644@@ -45,7 +45,6 @@ class AutoThrottle: def _spider_opened(self, spider: Spider) -> None: self.mindelay = self._min_delay() self.maxdelay = self._max_delay()- assert self.crawler.engine self.crawler.engine.downloader._delay = self._start_delay() def _min_delay(self) -> float:@@ -98,7 +97,6 @@ class AutoThrottle: key: str | None = request.meta.get("download_slot") if key is None: return None, None- assert self.crawler.engine return key, self.crawler.engine.downloader.slots.get(key) def _adjust_delay(self, slot: Slot, latency: float, response: Response) -> None:scrapy/pipelines/files.pyindex 55a3676e5..e666f4ddd 100644@@ -697,9 +697,9 @@ class FilesPipeline(MediaPipeline): } def inc_stats(self, status: str) -> None:- assert self.crawler.stats- self.crawler.stats.inc_value("file_count")- self.crawler.stats.inc_value(f"file_status_count/{status}")+ stats = self.crawler.stats+ stats.inc_value("file_count")+ stats.inc_value(f"file_status_count/{status}") async def _file_downloaded( self,scrapy/pipelines/media.pyindex 764f82a78..d4025bf3b 100644@@ -100,7 +100,6 @@ class MediaPipeline(ABC): stacklevel=2, ) self.crawler: Crawler = crawler- assert crawler.request_fingerprinter self._fingerprinter: RequestFingerprinterProtocol = ( crawler.request_fingerprinter )@@ -228,7 +227,6 @@ class MediaPipeline(ABC): ) -> FileInfo: try: self._modify_media_request(request)- assert self.crawler.engine response = await self.crawler.engine.download_async(request) return await ensure_awaitable( self.media_downloaded(response, request, info, item=item)scrapy/pqueues.pyindex 8e9783f5a..f4c35c0fe 100644@@ -265,7 +265,6 @@ class ScrapyPriorityQueue: class DownloaderInterface: def __init__(self, crawler: Crawler):- assert crawler.engine self.downloader: Downloader = crawler.engine.downloader def stats(self, possible_slots: Iterable[str]) -> list[tuple[int, str]]:scrapy/shell.pyindex dfea00c46..8f78af571 100644@@ -193,7 +193,6 @@ class Shell: """ if not self.spider: await self._open_spider(spider)- assert self.crawler.engine is not None # send the request to the engine self.crawler.engine.crawl(request) # this will fire when the request callback runs (via the callback hijacking in _request_deferred())@@ -204,7 +203,6 @@ class Shell: spider = self.crawler.spider or self.crawler._create_spider() self.crawler.spider = spider- assert self.crawler.engine await self.crawler.engine.open_spider_async(close_if_idle=False) self.spider = spider scrapy/spidermiddlewares/depth.pyindex 1781543e3..49683168e 100644@@ -70,7 +70,6 @@ class DepthMiddleware(BaseSpiderMiddleware): maxdepth = settings.getint("DEPTH_LIMIT") verbose = settings.getbool("DEPTH_STATS_VERBOSE") prio = settings.getint("DEPTH_PRIORITY")- assert crawler.stats o = cls(maxdepth, crawler.stats, verbose, prio) o.crawler = crawler return oscrapy/spidermiddlewares/httperror.pyindex 156b73e7e..116a02733 100644@@ -78,9 +78,9 @@ class HttpErrorMiddleware: self, response: Response, exception: Exception, spider: Spider | None = None ) -> Iterable[Any] | None: if isinstance(exception, HttpError):- assert self.crawler.stats- self.crawler.stats.inc_value("httperror/response_ignored_count")- self.crawler.stats.inc_value(+ stats = self.crawler.stats+ stats.inc_value("httperror/response_ignored_count")+ stats.inc_value( f"httperror/response_ignored_status_count/{response.status}" ) logger.info(scrapy/spidermiddlewares/urllength.pyindex f325ce7a0..86bdd2ed6 100644@@ -48,6 +48,5 @@ class UrlLengthMiddleware(BaseSpiderMiddleware): {"maxlength": self.maxlength, "url": request.url}, extra={"spider": self.crawler.spider}, )- assert self.crawler.stats self.crawler.stats.inc_value("urllength/request_ignored_count") return Nonescrapy/utils/log.pyindex ee357c031..bfa39169f 100644@@ -239,7 +239,6 @@ class LogCounterHandler(logging.Handler): def emit(self, record: logging.LogRecord) -> None: sname = f"log_count/{record.levelname}"- assert self.crawler.stats self.crawler.stats.inc_value(sname) tests/test_crawler.pyindex 358f20ed7..17ca02dea 100644@@ -74,6 +74,31 @@ class TestCrawler: assert not settings.frozen assert crawler.settings.frozen + @pytest.mark.parametrize(+ "attr",+ ["extensions", "logformatter", "request_fingerprinter", "stats"],+ )+ def test_late_attr_before_apply_settings(self, attr: str) -> None:+ crawler = get_raw_crawler(DefaultSpider)+ with pytest.raises(RuntimeError, match=rf"Crawler\.{attr} is not set yet"):+ getattr(crawler, attr)+ crawler._apply_settings()+ assert getattr(crawler, attr) is not None++ @pytest.mark.parametrize(+ "attr",+ ["engine", "extensions", "logformatter", "request_fingerprinter", "stats"],+ )+ def test_late_attr_on_class(self, attr: str) -> None:+ # Introspection tools such as help() read these off the class.+ assert getattr(Crawler, attr) is getattr(Crawler, attr)++ def test_late_attr_engine_before_crawl(self) -> None:+ crawler = get_raw_crawler(DefaultSpider)+ crawler._apply_settings()+ with pytest.raises(RuntimeError, match=r"Crawler\.engine is not set yet"):+ _ = crawler.engine+ @pytest.mark.parametrize( ("attr", "setting"), [