Backtestsscrapyrun-20261025-211112#7766
Move internal scrapy.utils modules to internal names
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
The issue's remaining undeprecated modules (given base already has _compression, _versions, _shell and ignores for benchserver/console/gz/job/ssl/versions) are datatypes, display, ftp, ossignal, sitemap and template. Patch B deprecates all six, moving each to an underscored module, updating every internal call site (headers, resolver, misc, media, shell, fetch, files, etc.) and the shims emit ScrapyDeprecationWarning. Patch A omits datatypes entirely, leaving one of the issue's listed modules untouched; the independent test for scrapy.utils.datatypes deprecation fails on A and passes on B, and the patch text confirms A has no datatypes change. On tests, B adds tests/test_utils_deprecated_modules.py, which verifies each shim warns, re-exports the exact names, and that Scrapy's own modules never import the deprecated paths in a fresh subprocess; these would fail before and pass after. A only retargets existing test imports and adds no coverage for the deprecation behaviour. B also sweeps news.rst cross-references to literals so the docs build won't reference deprecated modules, which is consistent with the issue's 'not mentioned in the docs' criterion. Craft nits: B's new underscore imports are placed out of isort order in several files (e.g. _ossignal after misc, _ftp after defer), which A gets right, and A's rename to _colorize with a public tty_supports_color is a reasonable touch. Those are minor compared to A's missing module and lack of tests. B is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
20m 51s
Wall-clock time in the container
Full test suite
No new failures
3,002 passed · 391 skipped · 3,403 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
1,152vs 730 merged
Changed lines, ours vs merged
The issue
View on GitHubFollow-up to https://github.com/scrapy/scrapy/pull/7007#discussion_r2775510982
Idea: I assume that most of (especially older) utils were moved there to organize the Scrapy code, not to allow users to use those functions. Even if we currently don't plan to remove or break them it's useful to be able to freely do that in the future.
For the first pass the criteria are:
- looking not at specific functions/classes but at whole modules;
- a module must not have names mentioned in the docs;
- deprecated names are ignored as if they don't exist;
- undocumented names that are definitely used by e.g. scrapy-poet or 3rd-party code that I could find or look like they may be useful to other users are for now treated as to be kept.
This list is already huge:
benchserver- class
Root consoleget_shell_embed_funcstart_python_console- const
DEFAULT_PYTHON_SHELLS datatypes- class
CaseInsensitiveDict - class
LocalCache - class
LocalWeakReferencedCache - class
SequenceExclude displaypformatpprintftpftp_makedirs_cwdftp_store_filegzgunzipgzip_magic_numberjobjob_dirossignalinstall_shutdown_handlerssitemapsitemap_urls_from_robots- class
Sitemap sslget_openssl_versiontemplaterender_templatefilestring_camelcase- const
CAMELCASE_INVALID_CHARS versionsget_versions
I propose deprecating all of these, moving to underscored utils modules or next to call sites for utils that didn't prove to be worth a shared utils location.
The two patches
docs/news.rstindex 299358c8f..e7664709a 100644@@ -249,7 +249,7 @@ Bug fixes points to can be imported. (:gh:`4780`, :gh:`8042`) -- :class:`scrapy.utils.datatypes.LocalCache` no longer evicts the oldest item+- ``scrapy.utils.datatypes.LocalCache`` no longer evicts the oldest item when updating an existing one. (:gh:`8113`) @@ -535,7 +535,7 @@ Deprecation removals - ``scrapy.utils.datatypes.CaselessDict``, deprecated since Scrapy 2.10.0, is removed. Use- :class:`~scrapy.utils.datatypes.CaseInsensitiveDict` instead.+ ``scrapy.utils.datatypes.CaseInsensitiveDict`` instead. (:gh:`5146`, :gh:`8023`) Deprecations@@ -1371,7 +1371,7 @@ Bug fixes looking for "nofollow" strings. (:gh:`7632`) -- Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing+- Fixed an exception in ``scrapy.utils.sitemap.Sitemap`` when parsing some malformed sitemaps. (:gh:`7686`, :gh:`7687`) @@ -1889,8 +1889,8 @@ Deprecations (:gh:`7353`, :gh:`7391`) - Passing :class:`str` instead of :class:`bytes` to- :class:`scrapy.utils.sitemap.Sitemap` and- :func:`scrapy.utils.sitemap.sitemap_urls_from_robots` is deprecated.+ ``scrapy.utils.sitemap.Sitemap`` and+ ``scrapy.utils.sitemap.sitemap_urls_from_robots()`` is deprecated. (:gh:`7007`) - ``scrapy.utils.misc.walk_modules()`` is deprecated. You can use@@ -2020,7 +2020,7 @@ Improvements (:gh:`6859`, :gh:`7353`) - Improved memory usage of :class:`~scrapy.spiders.sitemap.SitemapSpider` and- :class:`scrapy.utils.sitemap.Sitemap`.+ ``scrapy.utils.sitemap.Sitemap``. (:gh:`3529`, :gh:`7007`) - Improved the scheduling behavior of@@ -4608,8 +4608,8 @@ Deprecation removals Deprecations ~~~~~~~~~~~~ -- :class:`scrapy.utils.datatypes.CaselessDict` is deprecated, use- :class:`scrapy.utils.datatypes.CaseInsensitiveDict` instead.+- ``scrapy.utils.datatypes.CaselessDict`` is deprecated, use+ ``scrapy.utils.datatypes.CaseInsensitiveDict`` instead. (:gh:`5146`) - Passing the ``custom`` argument to@@ -4665,8 +4665,8 @@ Bug fixes - Added the missing :meth:`scrapy.settings.BaseSettings.pop` method. (:gh:`5959`, :gh:`5960`, :gh:`5963`) -- Added :class:`~scrapy.utils.datatypes.CaseInsensitiveDict` as a replacement- for :class:`~scrapy.utils.datatypes.CaselessDict` that fixes some API+- Added ``scrapy.utils.datatypes.CaseInsensitiveDict`` as a replacement+ for ``scrapy.utils.datatypes.CaselessDict`` that fixes some API inconsistencies. (:gh:`5146`) Documentation@@ -6576,7 +6576,7 @@ Bug fixes folders and files have the right permissions (:gh:`4604`) * Fix a :exc:`KeyError` exception being sometimes raised from- :class:`scrapy.utils.datatypes.LocalWeakReferencedCache` (:gh:`4597`,+ ``scrapy.utils.datatypes.LocalWeakReferencedCache`` (:gh:`4597`, :gh:`4599`) * When :setting:`FEEDS` defines multiple URIs, log messages about items being@@ -6708,7 +6708,7 @@ Deprecation removals :setting:`SPIDER_LOADER_CLASS` with your settings instead (:gh:`4398`) * The ``MultiValueDict``, ``MultiValueDictKeyError``, and ``SiteNode``- classes have been removed from :mod:`scrapy.utils.datatypes`+ classes have been removed from ``scrapy.utils.datatypes`` (:gh:`4400`) @@ -7064,7 +7064,7 @@ Bug fixes * Z shell auto-completion now looks for ``.html`` files, not ``.http`` files, and covers the ``-h`` command-line switch (:gh:`4122`, :gh:`4291`) -* Adding items to a :class:`scrapy.utils.datatypes.LocalCache` object+* Adding items to a ``scrapy.utils.datatypes.LocalCache`` object without a ``limit`` defined no longer raises a :exc:`TypeError` exception (:gh:`4123`) scrapy/commands/fetch.pyindex 809f6918d..c345e297c 100644@@ -9,7 +9,7 @@ from w3lib.url import is_url from scrapy.commands import ScrapyCommand from scrapy.exceptions import UsageError from scrapy.http import Request, Response-from scrapy.utils.datatypes import SequenceExclude+from scrapy.utils._datatypes import SequenceExclude from scrapy.utils.spider import DefaultSpider, spidercls_for_request if TYPE_CHECKING:scrapy/commands/genspider.pyindex c32a01395..79c25cd98 100644@@ -12,7 +12,7 @@ from scrapy.commands import ScrapyCommand from scrapy.commands.edit import _edit_file from scrapy.exceptions import UsageError from scrapy.spiderloader import get_spider_loader-from scrapy.utils.template import render_templatefile, string_camelcase+from scrapy.utils._template import render_templatefile, string_camelcase if TYPE_CHECKING: import argparsescrapy/commands/parse.pyindex d50075fb5..19f71e2d2 100644@@ -13,7 +13,7 @@ from w3lib.url import is_url from scrapy.commands import BaseRunSpiderCommand from scrapy.exceptions import UsageError from scrapy.http import Request, Response-from scrapy.utils import display+from scrapy.utils import _display as display from scrapy.utils.asyncgen import collect_asyncgen from scrapy.utils.defer import _schedule_coro, aiter_errback, deferred_from_coro from scrapy.utils.log import failure_to_exc_infoscrapy/commands/startproject.pyindex 5661a0a48..497c24e9e 100644@@ -11,7 +11,7 @@ from typing import TYPE_CHECKING, Any, ClassVar import scrapy from scrapy.commands import ScrapyCommand from scrapy.exceptions import UsageError-from scrapy.utils.template import render_templatefile, string_camelcase+from scrapy.utils._template import render_templatefile, string_camelcase if TYPE_CHECKING: import argparsescrapy/crawler.pyindex 0c2830c4c..65df8584e 100644@@ -30,7 +30,7 @@ from scrapy.utils.log import ( log_scrapy_info, ) from scrapy.utils.misc import build_from_crawler, load_object-from scrapy.utils.ossignal import install_shutdown_handlers, signal_names+from scrapy.utils._ossignal import install_shutdown_handlers, signal_names from scrapy.utils.reactor import ( _asyncio_reactor_path, _is_asyncio_reactor_installed,scrapy/extensions/feedexport.pyindex 320106705..60f5c797d 100644@@ -31,7 +31,7 @@ from scrapy.utils.asyncio import is_asyncio_available, run_in_thread from scrapy.utils.boto import _get_max_pool_connections from scrapy.utils.conf import feed_complete_default_values_from_settings from scrapy.utils.defer import deferred_from_coro, ensure_awaitable-from scrapy.utils.ftp import ftp_store_file+from scrapy.utils._ftp import ftp_store_file from scrapy.utils.misc import build_from_crawler, load_object from scrapy.utils.python import without_none_values scrapy/http/headers.pyindex dead57cdb..e3509ef72 100644@@ -5,7 +5,7 @@ from typing import TYPE_CHECKING, Any, TypeAlias, cast from w3lib.http import headers_dict_to_raw -from scrapy.utils.datatypes import CaseInsensitiveDict+from scrapy.utils._datatypes import CaseInsensitiveDict from scrapy.utils.python import to_unicode if TYPE_CHECKING:scrapy/pipelines/files.pyindex 9d67e98bf..60f96ebea 100644@@ -36,9 +36,9 @@ from scrapy.pipelines.media import ( ) from scrapy.utils.asyncio import run_in_thread from scrapy.utils.boto import _get_max_pool_connections, is_botocore_available-from scrapy.utils.datatypes import CaseInsensitiveDict+from scrapy.utils._datatypes import CaseInsensitiveDict from scrapy.utils.defer import deferred_from_coro, ensure_awaitable-from scrapy.utils.ftp import ftp_store_file+from scrapy.utils._ftp import ftp_store_file from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.log import failure_to_exc_info from scrapy.utils.python import to_bytesscrapy/pipelines/media.pyindex 41b2e17c9..b5742acd4 100644@@ -15,7 +15,7 @@ from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.http.request import NO_CALLBACK, Request from scrapy.utils._deps_compat import TWISTED_FAILURE_HAS_STACK from scrapy.utils.asyncio import is_asyncio_available-from scrapy.utils.datatypes import SequenceExclude+from scrapy.utils._datatypes import SequenceExclude from scrapy.utils.decorators import _warn_spider_arg from scrapy.utils.defer import ( _process_pending_io,scrapy/resolver.pyindex 0e9775fd1..0f1631fba 100644@@ -14,7 +14,7 @@ from twisted.internet.interfaces import ( ) from zope.interface.declarations import implementer, provider -from scrapy.utils.datatypes import LocalCache+from scrapy.utils._datatypes import LocalCache if TYPE_CHECKING: from collections.abc import Sequencescrapy/shell.pyindex 294b43893..a04bf638f 100644@@ -27,7 +27,7 @@ from scrapy.settings import Settings from scrapy.spiders import Spider from scrapy.utils._shell import DEFAULT_PYTHON_SHELLS, start_python_console from scrapy.utils.conf import get_config-from scrapy.utils.datatypes import SequenceExclude+from scrapy.utils._datatypes import SequenceExclude from scrapy.utils.defer import deferred_f_from_coro_f, maybe_deferred_to_future from scrapy.utils.misc import load_object from scrapy.utils.response import open_in_browserscrapy/spiders/sitemap.pyindex 92acef9d3..4e73d247d 100644@@ -14,7 +14,7 @@ from scrapy.utils._compression import ( gunzip, gzip_magic_number, )-from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots+from scrapy.utils._sitemap import Sitemap, sitemap_urls_from_robots if TYPE_CHECKING: # typing.Self requires Python 3.11scrapy/utils/_datatypes.pynew file mode 100644index 000000000..5598fe9e4@@ -0,0 +1,138 @@+"""+This module contains data types used by Scrapy which are not included in the+Python Standard Library.+"""++from __future__ import annotations++import collections+import contextlib+import weakref+from collections import OrderedDict+from typing import TYPE_CHECKING, Any, TypeVar, cast++if TYPE_CHECKING:+ from collections.abc import Container++ # typing.Self requires Python 3.11+ from typing_extensions import Self+++_KT = TypeVar("_KT")+_VT = TypeVar("_VT")+++class CaseInsensitiveDict(collections.UserDict[str | bytes, Any]):+ """A dict-like structure that accepts strings or bytes+ as keys and allows case-insensitive lookups.+ """++ def __init__(self, *args: Any, **kwargs: Any) -> None:+ self._keys: dict[str | bytes, Any] = {}+ super().__init__(*args, **kwargs)++ def __getitem__(self, key: str | bytes) -> Any:+ normalized_key = self._normkey(key)+ return super().__getitem__(self._keys[normalized_key.lower()])++ def __setitem__(self, key: str | bytes, value: Any) -> None:+ normalized_key = self._normkey(key)+ try:+ lower_key = self._keys[normalized_key.lower()]+ del self[lower_key]+ except KeyError:+ pass+ super().__setitem__(normalized_key, self._normvalue(value))+ self._keys[normalized_key.lower()] = normalized_key++ def __delitem__(self, key: str | bytes) -> None:+ normalized_key = self._normkey(key)+ stored_key = self._keys.pop(normalized_key.lower())+ super().__delitem__(stored_key)++ def __contains__(self, key: str | bytes) -> bool: # type: ignore[override]+ normalized_key = self._normkey(key)+ return normalized_key.lower() in self._keys++ def __repr__(self) -> str:+ return f"<{self.__class__.__name__}: {super().__repr__()}>"++ # UserDict.copy() shallow-copies the instance, which would share self._keys+ # between the copy and the original.+ def __copy__(self) -> Self:+ new = self.__class__()+ new.data = self.data.copy()+ new._keys = self._keys.copy()+ return new++ copy = __copy__++ # UserDict.__ior__ updates self.data directly, which would leave self._keys+ # out of date.+ def __ior__(self, other: Any) -> Self: # type: ignore[override,misc]+ self.update(other)+ return self++ def _normkey(self, key: str | bytes) -> str | bytes:+ return key++ def _normvalue(self, value: Any) -> Any:+ return value+++class LocalCache(OrderedDict[_KT, _VT]):+ """Dictionary with a finite number of keys.++ Older items expires first.+ """++ def __init__(self, limit: int | None = None):+ super().__init__()+ self.limit: int | None = limit++ def __setitem__(self, key: _KT, value: _VT) -> None:+ if self.limit is not None:+ if self.limit == 0:+ return+ if key not in self:+ while len(self) >= self.limit:+ self.popitem(last=False)+ super().__setitem__(key, value)+++class LocalWeakReferencedCache(weakref.WeakKeyDictionary[_KT, _VT | None]):+ """+ A weakref.WeakKeyDictionary implementation that uses LocalCache as its+ underlying data structure, making it ordered and capable of being size-limited.++ Useful for memoization, while avoiding keeping received+ arguments in memory only because of the cached references.++ Note: like LocalCache and unlike weakref.WeakKeyDictionary,+ it cannot be instantiated with an initial dictionary.+ """++ def __init__(self, limit: int | None = None):+ super().__init__()+ self.data: LocalCache[_KT, _VT] = LocalCache(limit=limit)++ def __setitem__(self, key: _KT, value: _VT | None) -> None:+ # if raised, key is not weak-referenceable, skip caching+ with contextlib.suppress(TypeError):+ super().__setitem__(key, value)++ def __getitem__(self, key: _KT) -> _VT | None:+ try:+ return cast("_VT", super().__getitem__(key))+ except (TypeError, KeyError):+ return None # key is either not weak-referenceable or not cached+++class SequenceExclude:+ """Object to test if an item is NOT within some sequence."""++ def __init__(self, seq: Container[Any]):+ self.seq: Container[Any] = seq++ def __contains__(self, item: Any) -> bool:+ return item not in self.seqscrapy/utils/_display.pynew file mode 100644index 000000000..08e89dc0d@@ -0,0 +1,51 @@+"""+pprint and pformat wrappers with colorization support+"""++import ctypes+import platform+import sys+from pprint import pformat as pformat_+from typing import Any++from packaging.version import Version as parse_version+++def _enable_windows_terminal_processing() -> bool:+ # https://stackoverflow.com/a/36760881+ kernel32 = ctypes.windll.kernel32 # type: ignore[attr-defined]+ return bool(kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7))+++def _tty_supports_color() -> bool:+ if sys.platform != "win32":+ return True++ if parse_version(platform.version()) < parse_version("10.0.14393"):+ return True++ # Windows >= 10.0.14393 interprets ANSI escape sequences providing terminal+ # processing is enabled.+ return _enable_windows_terminal_processing()+++def _colorize(text: str, colorize: bool = True) -> str:+ # pylint: disable=no-name-in-module+ if not colorize or not sys.stdout.isatty() or not _tty_supports_color():+ return text+ try:+ from pygments import highlight # noqa: PLC0415+ except ImportError:+ return text+ from pygments.formatters import TerminalFormatter # noqa: PLC0415+ from pygments.lexers import PythonLexer # noqa: PLC0415++ return highlight(text, PythonLexer(), TerminalFormatter())+++def pformat(obj: Any, *args: Any, **kwargs: Any) -> str:+ return _colorize(pformat_(obj), kwargs.pop("colorize", True))+++def pprint(obj: Any, *args: Any, **kwargs: Any) -> None:+ print(pformat(obj, *args, **kwargs))scrapy/utils/_ftp.pynew file mode 100644index 000000000..f08e5303d@@ -0,0 +1,53 @@+import posixpath+from contextlib import closing+from ftplib import FTP, FTP_TLS, error_perm+from posixpath import dirname+from ssl import create_default_context+from typing import IO+++def ftp_makedirs_cwd(ftp: FTP, path: str, first_call: bool = True) -> None:+ """Set the current directory of the FTP connection given in the ``ftp``+ argument (as a ftplib.FTP object), creating all parent directories if they+ don't exist. The ftplib.FTP object must be already connected and logged in.+ """+ try:+ ftp.cwd(path)+ except error_perm:+ ftp_makedirs_cwd(ftp, dirname(path), False)+ ftp.mkd(path)+ if first_call:+ ftp.cwd(path)+++def ftp_store_file(+ *,+ path: str,+ file: IO[bytes],+ host: str,+ port: int,+ username: str,+ password: str,+ use_active_mode: bool = False,+ overwrite: bool = True,+ tls: bool = False,+) -> None:+ """Opens a FTP connection with passed credentials, sets current directory+ to the directory extracted from given path, then uploads the file to server.++ If *tls* is ``True``, the connection is secured with TLS (FTPS), and the+ certificate of the server is verified.+ """+ ftp = FTP_TLS(context=create_default_context()) if tls else FTP()+ with ftp, closing(file):+ ftp.connect(host, port)+ ftp.login(username, password)+ if isinstance(ftp, FTP_TLS):+ ftp.prot_p()+ if use_active_mode:+ ftp.set_pasv(False)+ file.seek(0)+ dirname, filename = posixpath.split(path)+ ftp_makedirs_cwd(ftp, dirname)+ command = "STOR" if overwrite else "APPE"+ ftp.storbinary(f"{command} {filename}", file)scrapy/utils/_ossignal.pynew file mode 100644index 000000000..4ad8e5741@@ -0,0 +1,33 @@+from __future__ import annotations++import signal+from collections.abc import Callable+from types import FrameType+from typing import Any, TypeAlias++# copy of _HANDLER from typeshed/stdlib/signal.pyi+SignalHandlerT: TypeAlias = (+ Callable[[int, FrameType | None], Any] | int | signal.Handlers | None+)++signal_names: dict[int, str] = {member.value: member.name for member in signal.Signals}+++def install_shutdown_handlers(+ function: SignalHandlerT, override_sigint: bool = True+) -> None:+ """Install the given function as a signal handler for all common shutdown+ signals (such as SIGINT, SIGTERM, etc). If ``override_sigint`` is ``False`` the+ SIGINT handler won't be installed if there is already a handler in place+ (e.g. Pdb)+ """+ signal.signal(signal.SIGTERM, function)+ if (+ signal.getsignal(signal.SIGINT) # pylint: disable=comparison-with-callable+ == signal.default_int_handler+ or override_sigint+ ):+ signal.signal(signal.SIGINT, function)+ # Catch Ctrl-Break in windows+ if hasattr(signal, "SIGBREAK"):+ signal.signal(signal.SIGBREAK, function)scrapy/utils/_sitemap.pynew file mode 100644index 000000000..03b7bf3b1@@ -0,0 +1,136 @@+"""+Module for processing Sitemaps.++Note: The main purpose of this module is to provide support for the+SitemapSpider, its API is subject to change without notice.+"""++from __future__ import annotations++import warnings+from io import BytesIO, StringIO+from typing import TYPE_CHECKING, Any+from urllib.parse import urljoin++import lxml.etree++from scrapy.exceptions import ScrapyDeprecationWarning++if TYPE_CHECKING:+ from collections.abc import Iterable, Iterator+++class Sitemap:+ """Class to parse Sitemap (type=urlset) and Sitemap Index+ (type=sitemapindex) files"""++ __slots__ = ("type", "xmliter")++ def __init__(self, xmltext: str | bytes):+ if isinstance(xmltext, str):+ warnings.warn(+ "Passing `str` type as `xmltext` is deprecated, use `bytes`",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ xmltext = xmltext.encode()++ self.xmliter = lxml.etree.iterparse(+ BytesIO(xmltext),+ recover=True,+ remove_comments=True,+ resolve_entities=False,+ remove_blank_text=True,+ collect_ids=False,+ remove_pis=True,+ events=("start", "end"),+ )+ _, root = next(self.xmliter)+ self.type = self._get_tag_name(root)++ def __iter__(self) -> Iterator[dict[str, Any]]:+ for event, elem in self.xmliter:+ if event == "start":+ continue++ if self._get_tag_name(elem) not in {"url", "sitemap"}:+ continue++ if d := self._process_sitemap_element(elem):+ yield d++ def _process_sitemap_element(+ self, elem: lxml.etree._Element+ ) -> dict[str, Any] | None:+ d: dict[str, Any] = {}+ alternate: list[str] = []+ has_loc = False++ for el in elem:+ try:+ tag_name = self._get_tag_name(el)+ if not tag_name:+ continue++ if tag_name == "link":+ if href := el.get("href"):+ alternate.append(href)+ else:+ d[tag_name] = el.text.strip() if el.text else ""+ if not has_loc and tag_name == "loc":+ has_loc = True+ finally:+ el.clear()+ elem.clear()+ parent = elem.getparent()+ if parent is not None:+ while elem.getprevious() is not None:+ del parent[0]++ if not has_loc:+ return None++ if alternate:+ d["alternate"] = alternate++ return d++ @staticmethod+ def _get_tag_name(elem: lxml.etree._Element) -> str:+ tag = elem.tag+ if not isinstance(tag, str):+ return ""+ _, _, localname = tag.partition("}")+ return localname or tag+++def sitemap_urls_from_robots(+ robots_text: str | bytes,+ base_url: str | None = None,+) -> Iterable[str]:+ if isinstance(robots_text, bytes):+ for line in BytesIO(robots_text):+ if line.lstrip()[:8].lower() == b"sitemap:":+ try:+ url = line.partition(b":")[2].strip().decode()+ except UnicodeDecodeError:+ continue+ yield urljoin(base_url or "", url)++ else:+ warnings.warn(+ "Passing `str` type as `robots_text` is deprecated, use `bytes`",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ yield from _sitemap_urls_from_robots_str(robots_text, base_url)+++def _sitemap_urls_from_robots_str(+ robots_text: str,+ base_url: str | None = None,+) -> Iterable[str]:+ for line in StringIO(robots_text):+ if line.lstrip()[:8].lower() == "sitemap:":+ url = line.partition(":")[2].strip()+ yield urljoin(base_url or "", url)scrapy/utils/_template.pynew file mode 100644index 000000000..977d5a42a@@ -0,0 +1,41 @@+"""Helper functions for working with templates"""++from __future__ import annotations++import re+import string+from pathlib import Path+from typing import TYPE_CHECKING, Any++if TYPE_CHECKING:+ from os import PathLike+++def render_templatefile(path: str | PathLike[str], **kwargs: Any) -> None:+ path_obj = Path(path)+ raw = path_obj.read_text("utf8")++ content = string.Template(raw).substitute(**kwargs)++ render_path = path_obj.with_suffix("") if path_obj.suffix == ".tmpl" else path_obj++ if path_obj.suffix == ".tmpl":+ path_obj.rename(render_path)++ render_path.write_text(content, "utf8")+++CAMELCASE_INVALID_CHARS = re.compile(r"[^a-zA-Z\d]")+++def string_camelcase(string: str) -> str:+ """Convert a word to its CamelCase version and remove invalid chars++ >>> string_camelcase('lost-pound')+ 'LostPound'++ >>> string_camelcase('missing_images')+ 'MissingImages'++ """+ return CAMELCASE_INVALID_CHARS.sub("", string.title())scrapy/utils/datatypes.pyindex 5598fe9e4..7bdfea9ab 100644@@ -1,138 +1,23 @@-"""-This module contains data types used by Scrapy which are not included in the-Python Standard Library.-"""--from __future__ import annotations--import collections-import contextlib-import weakref-from collections import OrderedDict-from typing import TYPE_CHECKING, Any, TypeVar, cast--if TYPE_CHECKING:- from collections.abc import Container-- # typing.Self requires Python 3.11- from typing_extensions import Self---_KT = TypeVar("_KT")-_VT = TypeVar("_VT")---class CaseInsensitiveDict(collections.UserDict[str | bytes, Any]):- """A dict-like structure that accepts strings or bytes- as keys and allows case-insensitive lookups.- """-- def __init__(self, *args: Any, **kwargs: Any) -> None:- self._keys: dict[str | bytes, Any] = {}- super().__init__(*args, **kwargs)-- def __getitem__(self, key: str | bytes) -> Any:- normalized_key = self._normkey(key)- return super().__getitem__(self._keys[normalized_key.lower()])-- def __setitem__(self, key: str | bytes, value: Any) -> None:- normalized_key = self._normkey(key)- try:- lower_key = self._keys[normalized_key.lower()]- del self[lower_key]- except KeyError:- pass- super().__setitem__(normalized_key, self._normvalue(value))- self._keys[normalized_key.lower()] = normalized_key-- def __delitem__(self, key: str | bytes) -> None:- normalized_key = self._normkey(key)- stored_key = self._keys.pop(normalized_key.lower())- super().__delitem__(stored_key)-- def __contains__(self, key: str | bytes) -> bool: # type: ignore[override]- normalized_key = self._normkey(key)- return normalized_key.lower() in self._keys-- def __repr__(self) -> str:- return f"<{self.__class__.__name__}: {super().__repr__()}>"-- # UserDict.copy() shallow-copies the instance, which would share self._keys- # between the copy and the original.- def __copy__(self) -> Self:- new = self.__class__()- new.data = self.data.copy()- new._keys = self._keys.copy()- return new-- copy = __copy__-- # UserDict.__ior__ updates self.data directly, which would leave self._keys- # out of date.- def __ior__(self, other: Any) -> Self: # type: ignore[override,misc]- self.update(other)- return self-- def _normkey(self, key: str | bytes) -> str | bytes:- return key-- def _normvalue(self, value: Any) -> Any:- return value---class LocalCache(OrderedDict[_KT, _VT]):- """Dictionary with a finite number of keys.-- Older items expires first.- """-- def __init__(self, limit: int | None = None):- super().__init__()- self.limit: int | None = limit-- def __setitem__(self, key: _KT, value: _VT) -> None:- if self.limit is not None:- if self.limit == 0:- return- if key not in self:- while len(self) >= self.limit:- self.popitem(last=False)- super().__setitem__(key, value)---class LocalWeakReferencedCache(weakref.WeakKeyDictionary[_KT, _VT | None]):- """- A weakref.WeakKeyDictionary implementation that uses LocalCache as its- underlying data structure, making it ordered and capable of being size-limited.-- Useful for memoization, while avoiding keeping received- arguments in memory only because of the cached references.-- Note: like LocalCache and unlike weakref.WeakKeyDictionary,- it cannot be instantiated with an initial dictionary.- """-- def __init__(self, limit: int | None = None):- super().__init__()- self.data: LocalCache[_KT, _VT] = LocalCache(limit=limit)-- def __setitem__(self, key: _KT, value: _VT | None) -> None:- # if raised, key is not weak-referenceable, skip caching- with contextlib.suppress(TypeError):- super().__setitem__(key, value)-- def __getitem__(self, key: _KT) -> _VT | None:- try:- return cast("_VT", super().__getitem__(key))- except (TypeError, KeyError):- return None # key is either not weak-referenceable or not cached---class SequenceExclude:- """Object to test if an item is NOT within some sequence."""-- def __init__(self, seq: Container[Any]):- self.seq: Container[Any] = seq-- def __contains__(self, item: Any) -> bool:- return item not in self.seq+# pragma: no file cover+import warnings++from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._datatypes import (+ CaseInsensitiveDict,+ LocalCache,+ LocalWeakReferencedCache,+ SequenceExclude,+)++warnings.warn(+ "The scrapy.utils.datatypes module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "CaseInsensitiveDict",+ "LocalCache",+ "LocalWeakReferencedCache",+ "SequenceExclude",+]scrapy/utils/display.pyindex 08e89dc0d..573d077cd 100644@@ -1,51 +1,16 @@-"""-pprint and pformat wrappers with colorization support-"""--import ctypes-import platform-import sys-from pprint import pformat as pformat_-from typing import Any--from packaging.version import Version as parse_version---def _enable_windows_terminal_processing() -> bool:- # https://stackoverflow.com/a/36760881- kernel32 = ctypes.windll.kernel32 # type: ignore[attr-defined]- return bool(kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7))---def _tty_supports_color() -> bool:- if sys.platform != "win32":- return True-- if parse_version(platform.version()) < parse_version("10.0.14393"):- return True-- # Windows >= 10.0.14393 interprets ANSI escape sequences providing terminal- # processing is enabled.- return _enable_windows_terminal_processing()---def _colorize(text: str, colorize: bool = True) -> str:- # pylint: disable=no-name-in-module- if not colorize or not sys.stdout.isatty() or not _tty_supports_color():- return text- try:- from pygments import highlight # noqa: PLC0415- except ImportError:- return text- from pygments.formatters import TerminalFormatter # noqa: PLC0415- from pygments.lexers import PythonLexer # noqa: PLC0415-- return highlight(text, PythonLexer(), TerminalFormatter())---def pformat(obj: Any, *args: Any, **kwargs: Any) -> str:- return _colorize(pformat_(obj), kwargs.pop("colorize", True))---def pprint(obj: Any, *args: Any, **kwargs: Any) -> None:- print(pformat(obj, *args, **kwargs))+# pragma: no file cover+import warnings++from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._display import pformat, pprint++warnings.warn(+ "The scrapy.utils.display module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "pformat",+ "pprint",+]scrapy/utils/ftp.pyindex f08e5303d..5bed01a3a 100644@@ -1,53 +1,16 @@-import posixpath-from contextlib import closing-from ftplib import FTP, FTP_TLS, error_perm-from posixpath import dirname-from ssl import create_default_context-from typing import IO+# pragma: no file cover+import warnings +from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._ftp import ftp_makedirs_cwd, ftp_store_file -def ftp_makedirs_cwd(ftp: FTP, path: str, first_call: bool = True) -> None:- """Set the current directory of the FTP connection given in the ``ftp``- argument (as a ftplib.FTP object), creating all parent directories if they- don't exist. The ftplib.FTP object must be already connected and logged in.- """- try:- ftp.cwd(path)- except error_perm:- ftp_makedirs_cwd(ftp, dirname(path), False)- ftp.mkd(path)- if first_call:- ftp.cwd(path)+warnings.warn(+ "The scrapy.utils.ftp module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+) --def ftp_store_file(- *,- path: str,- file: IO[bytes],- host: str,- port: int,- username: str,- password: str,- use_active_mode: bool = False,- overwrite: bool = True,- tls: bool = False,-) -> None:- """Opens a FTP connection with passed credentials, sets current directory- to the directory extracted from given path, then uploads the file to server.-- If *tls* is ``True``, the connection is secured with TLS (FTPS), and the- certificate of the server is verified.- """- ftp = FTP_TLS(context=create_default_context()) if tls else FTP()- with ftp, closing(file):- ftp.connect(host, port)- ftp.login(username, password)- if isinstance(ftp, FTP_TLS):- ftp.prot_p()- if use_active_mode:- ftp.set_pasv(False)- file.seek(0)- dirname, filename = posixpath.split(path)- ftp_makedirs_cwd(ftp, dirname)- command = "STOR" if overwrite else "APPE"- ftp.storbinary(f"{command} {filename}", file)+__all__ = [+ "ftp_makedirs_cwd",+ "ftp_store_file",+]scrapy/utils/log.pyindex bb4254f0c..f0b5f1812 100644@@ -17,7 +17,7 @@ import scrapy from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.settings import Settings from scrapy.utils._versions import get_versions-from scrapy.utils.display import _tty_supports_color+from scrapy.utils._display import _tty_supports_color if TYPE_CHECKING: from types import TracebackTypescrapy/utils/misc.pyindex 338649b65..7692c22d3 100644@@ -16,7 +16,7 @@ from typing import TYPE_CHECKING, Any, ParamSpec, Protocol, TypeVar, cast, overl from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.item import Item-from scrapy.utils.datatypes import LocalWeakReferencedCache+from scrapy.utils._datatypes import LocalWeakReferencedCache if TYPE_CHECKING: from collections.abc import Callable, Iterable, Iteratorscrapy/utils/ossignal.pyindex 4ad8e5741..a0eb39a6c 100644@@ -1,33 +1,21 @@-from __future__ import annotations+# pragma: no file cover+import warnings -import signal-from collections.abc import Callable-from types import FrameType-from typing import Any, TypeAlias--# copy of _HANDLER from typeshed/stdlib/signal.pyi-SignalHandlerT: TypeAlias = (- Callable[[int, FrameType | None], Any] | int | signal.Handlers | None+from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._ossignal import (+ SignalHandlerT,+ install_shutdown_handlers,+ signal_names, ) -signal_names: dict[int, str] = {member.value: member.name for member in signal.Signals}-+warnings.warn(+ "The scrapy.utils.ossignal module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+) -def install_shutdown_handlers(- function: SignalHandlerT, override_sigint: bool = True-) -> None:- """Install the given function as a signal handler for all common shutdown- signals (such as SIGINT, SIGTERM, etc). If ``override_sigint`` is ``False`` the- SIGINT handler won't be installed if there is already a handler in place- (e.g. Pdb)- """- signal.signal(signal.SIGTERM, function)- if (- signal.getsignal(signal.SIGINT) # pylint: disable=comparison-with-callable- == signal.default_int_handler- or override_sigint- ):- signal.signal(signal.SIGINT, function)- # Catch Ctrl-Break in windows- if hasattr(signal, "SIGBREAK"):- signal.signal(signal.SIGBREAK, function)+__all__ = [+ "SignalHandlerT",+ "install_shutdown_handlers",+ "signal_names",+]scrapy/utils/sitemap.pyindex 03b7bf3b1..cc66f5d1b 100644@@ -1,136 +1,16 @@-"""-Module for processing Sitemaps.--Note: The main purpose of this module is to provide support for the-SitemapSpider, its API is subject to change without notice.-"""--from __future__ import annotations-+# pragma: no file cover import warnings-from io import BytesIO, StringIO-from typing import TYPE_CHECKING, Any-from urllib.parse import urljoin--import lxml.etree from scrapy.exceptions import ScrapyDeprecationWarning--if TYPE_CHECKING:- from collections.abc import Iterable, Iterator---class Sitemap:- """Class to parse Sitemap (type=urlset) and Sitemap Index- (type=sitemapindex) files"""-- __slots__ = ("type", "xmliter")-- def __init__(self, xmltext: str | bytes):- if isinstance(xmltext, str):- warnings.warn(- "Passing `str` type as `xmltext` is deprecated, use `bytes`",- ScrapyDeprecationWarning,- stacklevel=2,- )- xmltext = xmltext.encode()-- self.xmliter = lxml.etree.iterparse(- BytesIO(xmltext),- recover=True,- remove_comments=True,- resolve_entities=False,- remove_blank_text=True,- collect_ids=False,- remove_pis=True,- events=("start", "end"),- )- _, root = next(self.xmliter)- self.type = self._get_tag_name(root)-- def __iter__(self) -> Iterator[dict[str, Any]]:- for event, elem in self.xmliter:- if event == "start":- continue-- if self._get_tag_name(elem) not in {"url", "sitemap"}:- continue-- if d := self._process_sitemap_element(elem):- yield d-- def _process_sitemap_element(- self, elem: lxml.etree._Element- ) -> dict[str, Any] | None:- d: dict[str, Any] = {}- alternate: list[str] = []- has_loc = False-- for el in elem:- try:- tag_name = self._get_tag_name(el)- if not tag_name:- continue-- if tag_name == "link":- if href := el.get("href"):- alternate.append(href)- else:- d[tag_name] = el.text.strip() if el.text else ""- if not has_loc and tag_name == "loc":- has_loc = True- finally:- el.clear()- elem.clear()- parent = elem.getparent()- if parent is not None:- while elem.getprevious() is not None:- del parent[0]-- if not has_loc:- return None-- if alternate:- d["alternate"] = alternate-- return d-- @staticmethod- def _get_tag_name(elem: lxml.etree._Element) -> str:- tag = elem.tag- if not isinstance(tag, str):- return ""- _, _, localname = tag.partition("}")- return localname or tag---def sitemap_urls_from_robots(- robots_text: str | bytes,- base_url: str | None = None,-) -> Iterable[str]:- if isinstance(robots_text, bytes):- for line in BytesIO(robots_text):- if line.lstrip()[:8].lower() == b"sitemap:":- try:- url = line.partition(b":")[2].strip().decode()- except UnicodeDecodeError:- continue- yield urljoin(base_url or "", url)-- else:- warnings.warn(- "Passing `str` type as `robots_text` is deprecated, use `bytes`",- ScrapyDeprecationWarning,- stacklevel=2,- )- yield from _sitemap_urls_from_robots_str(robots_text, base_url)---def _sitemap_urls_from_robots_str(- robots_text: str,- base_url: str | None = None,-) -> Iterable[str]:- for line in StringIO(robots_text):- if line.lstrip()[:8].lower() == "sitemap:":- url = line.partition(":")[2].strip()- yield urljoin(base_url or "", url)+from scrapy.utils._sitemap import Sitemap, sitemap_urls_from_robots++warnings.warn(+ "The scrapy.utils.sitemap module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "Sitemap",+ "sitemap_urls_from_robots",+]scrapy/utils/template.pyindex 977d5a42a..cccaba9a5 100644@@ -1,41 +1,21 @@-"""Helper functions for working with templates"""--from __future__ import annotations--import re-import string-from pathlib import Path-from typing import TYPE_CHECKING, Any--if TYPE_CHECKING:- from os import PathLike---def render_templatefile(path: str | PathLike[str], **kwargs: Any) -> None:- path_obj = Path(path)- raw = path_obj.read_text("utf8")-- content = string.Template(raw).substitute(**kwargs)-- render_path = path_obj.with_suffix("") if path_obj.suffix == ".tmpl" else path_obj-- if path_obj.suffix == ".tmpl":- path_obj.rename(render_path)-- render_path.write_text(content, "utf8")---CAMELCASE_INVALID_CHARS = re.compile(r"[^a-zA-Z\d]")---def string_camelcase(string: str) -> str:- """Convert a word to its CamelCase version and remove invalid chars-- >>> string_camelcase('lost-pound')- 'LostPound'-- >>> string_camelcase('missing_images')- 'MissingImages'-- """- return CAMELCASE_INVALID_CHARS.sub("", string.title())+# pragma: no file cover+import warnings++from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._template import (+ CAMELCASE_INVALID_CHARS,+ render_templatefile,+ string_camelcase,+)++warnings.warn(+ "The scrapy.utils.template module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "CAMELCASE_INVALID_CHARS",+ "render_templatefile",+ "string_camelcase",+]tests/ignores.txtindex 75ae2d141..4c6fa1dd3 100644@@ -4,7 +4,13 @@ scrapy/interfaces.py scrapy/mail.py scrapy/utils/benchserver.py scrapy/utils/console.py+scrapy/utils/datatypes.py+scrapy/utils/display.py+scrapy/utils/ftp.py scrapy/utils/gz.py scrapy/utils/job.py+scrapy/utils/ossignal.py+scrapy/utils/sitemap.py scrapy/utils/ssl.py+scrapy/utils/template.py scrapy/utils/versions.pytests/test_utils_datatypes.pyindex 55bef53b2..26096911d 100644@@ -7,7 +7,7 @@ from typing import Any import pytest from scrapy.http import Request-from scrapy.utils.datatypes import (+from scrapy.utils._datatypes import ( CaseInsensitiveDict, LocalCache, LocalWeakReferencedCache,tests/test_utils_deprecated_modules.pynew file mode 100644index 000000000..a87693384@@ -0,0 +1,79 @@+import importlib+import subprocess+import sys++import pytest++from scrapy.exceptions import ScrapyDeprecationWarning+++@pytest.mark.parametrize(+ ("module_name", "names"),+ [+ (+ "datatypes",+ [+ "CaseInsensitiveDict",+ "LocalCache",+ "LocalWeakReferencedCache",+ "SequenceExclude",+ ],+ ),+ ("display", ["pformat", "pprint"]),+ ("ftp", ["ftp_makedirs_cwd", "ftp_store_file"]),+ ("ossignal", ["SignalHandlerT", "install_shutdown_handlers", "signal_names"]),+ ("sitemap", ["Sitemap", "sitemap_urls_from_robots"]),+ (+ "template",+ ["CAMELCASE_INVALID_CHARS", "render_templatefile", "string_camelcase"],+ ),+ ],+)+def test_deprecated_module(+ monkeypatch: pytest.MonkeyPatch, module_name: str, names: list[str]+) -> None:+ old_path = f"scrapy.utils.{module_name}"+ monkeypatch.delitem(sys.modules, old_path, raising=False)+ with pytest.warns(+ ScrapyDeprecationWarning, match=rf"The {old_path} module is deprecated\."+ ):+ old_module = importlib.import_module(old_path)+ new_module = importlib.import_module(f"scrapy.utils._{module_name}")+ assert sorted(old_module.__all__) == sorted(names)+ for name in names:+ assert getattr(old_module, name) is getattr(new_module, name)+++@pytest.mark.parametrize(+ "module_name",+ [+ "scrapy.cmdline",+ "scrapy.crawler",+ "scrapy.shell",+ "scrapy.commands.fetch",+ "scrapy.commands.genspider",+ "scrapy.commands.parse",+ "scrapy.commands.startproject",+ "scrapy.extensions.feedexport",+ "scrapy.pipelines.files",+ "scrapy.spiders.sitemap",+ ],+)+def test_no_deprecated_module_imports(module_name: str) -> None:+ """Scrapy itself must not import the deprecated modules, as that would+ show deprecation warnings to users."""+ code = (+ "import sys\n"+ f"import {module_name}\n"+ "deprecated = [m for m in ("+ + ", ".join(+ repr(f"scrapy.utils.{m}")+ for m in ("datatypes", "display", "ftp", "ossignal", "sitemap", "template")+ )+ + ") if m in sys.modules]\n"+ "assert not deprecated, deprecated\n"+ )+ result = subprocess.run(+ [sys.executable, "-c", code], capture_output=True, text=True, check=False+ )+ assert result.returncode == 0, result.stderrtests/test_utils_display.pyindex b8d15bcba..99d913e46 100644@@ -8,7 +8,7 @@ from unittest import mock import pytest -from scrapy.utils.display import _enable_windows_terminal_processing, pformat, pprint+from scrapy.utils._display import _enable_windows_terminal_processing, pformat, pprint if TYPE_CHECKING: from collections.abc import Mapping, Sequence@@ -59,7 +59,7 @@ def test_pformat_old_windows(isatty: mock.Mock, version: mock.Mock) -> None: @mock.patch("sys.platform", "win32")-@mock.patch("scrapy.utils.display._enable_windows_terminal_processing")+@mock.patch("scrapy.utils._display._enable_windows_terminal_processing") @mock.patch("platform.version") @mock.patch("sys.stdout.isatty") def test_pformat_windows_no_terminal_processing(@@ -72,7 +72,7 @@ def test_pformat_windows_no_terminal_processing( @mock.patch("sys.platform", "win32")-@mock.patch("scrapy.utils.display._enable_windows_terminal_processing")+@mock.patch("scrapy.utils._display._enable_windows_terminal_processing") @mock.patch("platform.version") @mock.patch("sys.stdout.isatty") def test_pformat_windows(tests/test_utils_ossignal.pyindex 4c8f7d0b4..4dabebb21 100644@@ -5,7 +5,7 @@ from typing import TYPE_CHECKING, Any import pytest -from scrapy.utils.ossignal import install_shutdown_handlers+from scrapy.utils._ossignal import install_shutdown_handlers if TYPE_CHECKING: from collections.abc import Generatortests/test_utils_sitemap.pyindex dda13006d..571ad4075 100644@@ -3,7 +3,7 @@ from __future__ import annotations import pytest from scrapy.exceptions import ScrapyDeprecationWarning-from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots+from scrapy.utils._sitemap import Sitemap, sitemap_urls_from_robots def test_sitemap():tests/test_utils_template.pyindex 076edf3db..fed314974 100644@@ -2,7 +2,7 @@ from __future__ import annotations from typing import TYPE_CHECKING -from scrapy.utils.template import render_templatefile+from scrapy.utils._template import render_templatefile if TYPE_CHECKING: from pathlib import Path
scrapy/commands/genspider.pyindex c32a01395..79c25cd98 100644@@ -12,7 +12,7 @@ from scrapy.commands import ScrapyCommand from scrapy.commands.edit import _edit_file from scrapy.exceptions import UsageError from scrapy.spiderloader import get_spider_loader-from scrapy.utils.template import render_templatefile, string_camelcase+from scrapy.utils._template import render_templatefile, string_camelcase if TYPE_CHECKING: import argparsescrapy/commands/parse.pyindex d50075fb5..4edadfa58 100644@@ -13,7 +13,7 @@ from w3lib.url import is_url from scrapy.commands import BaseRunSpiderCommand from scrapy.exceptions import UsageError from scrapy.http import Request, Response-from scrapy.utils import display+from scrapy.utils import _colorize from scrapy.utils.asyncgen import collect_asyncgen from scrapy.utils.defer import _schedule_coro, aiter_errback, deferred_from_coro from scrapy.utils.log import failure_to_exc_info@@ -170,7 +170,7 @@ class Command(BaseRunSpiderCommand): items = self.items.get(lvl, []) print("# Scraped Items ", "-" * 60)- display.pprint([ItemAdapter(x).asdict() for x in items], colorize=colour)+ _colorize.pprint([ItemAdapter(x).asdict() for x in items], colorize=colour) def print_requests(self, lvl: int | None = None, colour: bool = True) -> None: if lvl is not None:@@ -181,7 +181,7 @@ class Command(BaseRunSpiderCommand): requests = [] print("# Requests ", "-" * 65)- display.pprint(requests, colorize=colour)+ _colorize.pprint(requests, colorize=colour) def print_results(self, opts: argparse.Namespace) -> None: colour = not opts.nocolourscrapy/commands/startproject.pyindex 5661a0a48..497c24e9e 100644@@ -11,7 +11,7 @@ from typing import TYPE_CHECKING, Any, ClassVar import scrapy from scrapy.commands import ScrapyCommand from scrapy.exceptions import UsageError-from scrapy.utils.template import render_templatefile, string_camelcase+from scrapy.utils._template import render_templatefile, string_camelcase if TYPE_CHECKING: import argparsescrapy/crawler.pyindex 0c2830c4c..1f40d563b 100644@@ -20,6 +20,7 @@ from scrapy.extension import ExtensionManager from scrapy.settings import SETTINGS_PRIORITIES, Settings, overridden_settings from scrapy.signalmanager import SignalManager from scrapy.spiderloader import SpiderLoaderProtocol, get_spider_loader+from scrapy.utils._ossignals import install_shutdown_handlers, signal_names from scrapy.utils._stopmode import _normalize_stop_mode, _StopMode from scrapy.utils.defer import _DEFER_DELAY, deferred_from_coro, ensure_awaitable from scrapy.utils.log import (@@ -30,7 +31,6 @@ from scrapy.utils.log import ( log_scrapy_info, ) from scrapy.utils.misc import build_from_crawler, load_object-from scrapy.utils.ossignal import install_shutdown_handlers, signal_names from scrapy.utils.reactor import ( _asyncio_reactor_path, _is_asyncio_reactor_installed,scrapy/extensions/feedexport.pyindex 320106705..d3dc03c53 100644@@ -27,11 +27,11 @@ from zope.interface import Interface from scrapy import Spider, signals from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning from scrapy.extensions.postprocessing import PostProcessingManager+from scrapy.utils._ftp import ftp_store_file from scrapy.utils.asyncio import is_asyncio_available, run_in_thread from scrapy.utils.boto import _get_max_pool_connections from scrapy.utils.conf import feed_complete_default_values_from_settings from scrapy.utils.defer import deferred_from_coro, ensure_awaitable-from scrapy.utils.ftp import ftp_store_file from scrapy.utils.misc import build_from_crawler, load_object from scrapy.utils.python import without_none_values scrapy/pipelines/files.pyindex 9d67e98bf..b3c44dee9 100644@@ -34,11 +34,11 @@ from scrapy.pipelines.media import ( MediaPipeline, _MediaRequestFiltered, )+from scrapy.utils._ftp import ftp_store_file from scrapy.utils.asyncio import run_in_thread from scrapy.utils.boto import _get_max_pool_connections, is_botocore_available from scrapy.utils.datatypes import CaseInsensitiveDict from scrapy.utils.defer import deferred_from_coro, ensure_awaitable-from scrapy.utils.ftp import ftp_store_file from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.log import failure_to_exc_info from scrapy.utils.python import to_bytesscrapy/spiders/sitemap.pyindex 92acef9d3..4e73d247d 100644@@ -14,7 +14,7 @@ from scrapy.utils._compression import ( gunzip, gzip_magic_number, )-from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots+from scrapy.utils._sitemap import Sitemap, sitemap_urls_from_robots if TYPE_CHECKING: # typing.Self requires Python 3.11scrapy/utils/_colorize.pynew file mode 100644index 000000000..53aa685fc@@ -0,0 +1,47 @@+import ctypes+import platform+import sys+from pprint import pformat as pformat_+from typing import Any++from packaging.version import Version as parse_version+++def _enable_windows_terminal_processing() -> bool:+ # https://stackoverflow.com/a/36760881+ kernel32 = ctypes.windll.kernel32 # type: ignore[attr-defined]+ return bool(kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7))+++def tty_supports_color() -> bool:+ if sys.platform != "win32":+ return True++ if parse_version(platform.version()) < parse_version("10.0.14393"):+ return True++ # Windows >= 10.0.14393 interprets ANSI escape sequences providing terminal+ # processing is enabled.+ return _enable_windows_terminal_processing()+++def _colorize(text: str, colorize: bool = True) -> str:+ # pylint: disable=no-name-in-module+ if not colorize or not sys.stdout.isatty() or not tty_supports_color():+ return text+ try:+ from pygments import highlight # noqa: PLC0415+ except ImportError:+ return text+ from pygments.formatters import TerminalFormatter # noqa: PLC0415+ from pygments.lexers import PythonLexer # noqa: PLC0415++ return highlight(text, PythonLexer(), TerminalFormatter())+++def pformat(obj: object, *args: Any, **kwargs: Any) -> str:+ return _colorize(pformat_(obj), kwargs.pop("colorize", True))+++def pprint(obj: object, *args: Any, **kwargs: Any) -> None:+ print(pformat(obj, *args, **kwargs))scrapy/utils/_ftp.pynew file mode 100644index 000000000..f08e5303d@@ -0,0 +1,53 @@+import posixpath+from contextlib import closing+from ftplib import FTP, FTP_TLS, error_perm+from posixpath import dirname+from ssl import create_default_context+from typing import IO+++def ftp_makedirs_cwd(ftp: FTP, path: str, first_call: bool = True) -> None:+ """Set the current directory of the FTP connection given in the ``ftp``+ argument (as a ftplib.FTP object), creating all parent directories if they+ don't exist. The ftplib.FTP object must be already connected and logged in.+ """+ try:+ ftp.cwd(path)+ except error_perm:+ ftp_makedirs_cwd(ftp, dirname(path), False)+ ftp.mkd(path)+ if first_call:+ ftp.cwd(path)+++def ftp_store_file(+ *,+ path: str,+ file: IO[bytes],+ host: str,+ port: int,+ username: str,+ password: str,+ use_active_mode: bool = False,+ overwrite: bool = True,+ tls: bool = False,+) -> None:+ """Opens a FTP connection with passed credentials, sets current directory+ to the directory extracted from given path, then uploads the file to server.++ If *tls* is ``True``, the connection is secured with TLS (FTPS), and the+ certificate of the server is verified.+ """+ ftp = FTP_TLS(context=create_default_context()) if tls else FTP()+ with ftp, closing(file):+ ftp.connect(host, port)+ ftp.login(username, password)+ if isinstance(ftp, FTP_TLS):+ ftp.prot_p()+ if use_active_mode:+ ftp.set_pasv(False)+ file.seek(0)+ dirname, filename = posixpath.split(path)+ ftp_makedirs_cwd(ftp, dirname)+ command = "STOR" if overwrite else "APPE"+ ftp.storbinary(f"{command} {filename}", file)scrapy/utils/_ossignals.pynew file mode 100644index 000000000..4ad8e5741@@ -0,0 +1,33 @@+from __future__ import annotations++import signal+from collections.abc import Callable+from types import FrameType+from typing import Any, TypeAlias++# copy of _HANDLER from typeshed/stdlib/signal.pyi+SignalHandlerT: TypeAlias = (+ Callable[[int, FrameType | None], Any] | int | signal.Handlers | None+)++signal_names: dict[int, str] = {member.value: member.name for member in signal.Signals}+++def install_shutdown_handlers(+ function: SignalHandlerT, override_sigint: bool = True+) -> None:+ """Install the given function as a signal handler for all common shutdown+ signals (such as SIGINT, SIGTERM, etc). If ``override_sigint`` is ``False`` the+ SIGINT handler won't be installed if there is already a handler in place+ (e.g. Pdb)+ """+ signal.signal(signal.SIGTERM, function)+ if (+ signal.getsignal(signal.SIGINT) # pylint: disable=comparison-with-callable+ == signal.default_int_handler+ or override_sigint+ ):+ signal.signal(signal.SIGINT, function)+ # Catch Ctrl-Break in windows+ if hasattr(signal, "SIGBREAK"):+ signal.signal(signal.SIGBREAK, function)scrapy/utils/_sitemap.pynew file mode 100644index 000000000..2edf2d5ef@@ -0,0 +1,129 @@+from __future__ import annotations++import warnings+from io import BytesIO, StringIO+from typing import TYPE_CHECKING, Any+from urllib.parse import urljoin++import lxml.etree++from scrapy.exceptions import ScrapyDeprecationWarning++if TYPE_CHECKING:+ from collections.abc import Iterable, Iterator+++class Sitemap:+ """Class to parse Sitemap (type=urlset) and Sitemap Index+ (type=sitemapindex) files"""++ __slots__ = ("type", "xmliter")++ def __init__(self, xmltext: str | bytes):+ if isinstance(xmltext, str):+ warnings.warn(+ "Passing `str` type as `xmltext` is deprecated, use `bytes`",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ xmltext = xmltext.encode()++ self.xmliter = lxml.etree.iterparse(+ BytesIO(xmltext),+ recover=True,+ remove_comments=True,+ resolve_entities=False,+ remove_blank_text=True,+ collect_ids=False,+ remove_pis=True,+ events=("start", "end"),+ )+ _, root = next(self.xmliter)+ self.type = self._get_tag_name(root)++ def __iter__(self) -> Iterator[dict[str, Any]]:+ for event, elem in self.xmliter:+ if event == "start":+ continue++ if self._get_tag_name(elem) not in {"url", "sitemap"}:+ continue++ if d := self._process_sitemap_element(elem):+ yield d++ def _process_sitemap_element(+ self, elem: lxml.etree._Element+ ) -> dict[str, Any] | None:+ d: dict[str, Any] = {}+ alternate: list[str] = []+ has_loc = False++ for el in elem:+ try:+ tag_name = self._get_tag_name(el)+ if not tag_name:+ continue++ if tag_name == "link":+ if href := el.get("href"):+ alternate.append(href)+ else:+ d[tag_name] = el.text.strip() if el.text else ""+ if not has_loc and tag_name == "loc":+ has_loc = True+ finally:+ el.clear()+ elem.clear()+ parent = elem.getparent()+ if parent is not None:+ while elem.getprevious() is not None:+ del parent[0]++ if not has_loc:+ return None++ if alternate:+ d["alternate"] = alternate++ return d++ @staticmethod+ def _get_tag_name(elem: lxml.etree._Element) -> str:+ tag = elem.tag+ if not isinstance(tag, str):+ return ""+ _, _, localname = tag.partition("}")+ return localname or tag+++def sitemap_urls_from_robots(+ robots_text: str | bytes,+ base_url: str | None = None,+) -> Iterable[str]:+ if isinstance(robots_text, bytes):+ for line in BytesIO(robots_text):+ if line.lstrip()[:8].lower() == b"sitemap:":+ try:+ url = line.partition(b":")[2].strip().decode()+ except UnicodeDecodeError:+ continue+ yield urljoin(base_url or "", url)++ else:+ warnings.warn(+ "Passing `str` type as `robots_text` is deprecated, use `bytes`",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ yield from _sitemap_urls_from_robots_str(robots_text, base_url)+++def _sitemap_urls_from_robots_str(+ robots_text: str,+ base_url: str | None = None,+) -> Iterable[str]:+ for line in StringIO(robots_text):+ if line.lstrip()[:8].lower() == "sitemap:":+ url = line.partition(":")[2].strip()+ yield urljoin(base_url or "", url)scrapy/utils/_template.pynew file mode 100644index 000000000..55c006b39@@ -0,0 +1,39 @@+from __future__ import annotations++import re+import string+from pathlib import Path+from typing import TYPE_CHECKING, Any++if TYPE_CHECKING:+ from os import PathLike+++def render_templatefile(path: str | PathLike[str], **kwargs: Any) -> None:+ path_obj = Path(path)+ raw = path_obj.read_text("utf8")++ content = string.Template(raw).substitute(**kwargs)++ render_path = path_obj.with_suffix("") if path_obj.suffix == ".tmpl" else path_obj++ if path_obj.suffix == ".tmpl":+ path_obj.rename(render_path)++ render_path.write_text(content, "utf8")+++CAMELCASE_INVALID_CHARS = re.compile(r"[^a-zA-Z\d]")+++def string_camelcase(string: str) -> str:+ """Convert a word to its CamelCase version and remove invalid chars++ >>> string_camelcase('lost-pound')+ 'LostPound'++ >>> string_camelcase('missing_images')+ 'MissingImages'++ """+ return CAMELCASE_INVALID_CHARS.sub("", string.title())scrapy/utils/display.pyindex 08e89dc0d..88d219840 100644@@ -1,51 +1,16 @@-"""-pprint and pformat wrappers with colorization support-"""--import ctypes-import platform-import sys-from pprint import pformat as pformat_-from typing import Any--from packaging.version import Version as parse_version---def _enable_windows_terminal_processing() -> bool:- # https://stackoverflow.com/a/36760881- kernel32 = ctypes.windll.kernel32 # type: ignore[attr-defined]- return bool(kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7))---def _tty_supports_color() -> bool:- if sys.platform != "win32":- return True-- if parse_version(platform.version()) < parse_version("10.0.14393"):- return True-- # Windows >= 10.0.14393 interprets ANSI escape sequences providing terminal- # processing is enabled.- return _enable_windows_terminal_processing()---def _colorize(text: str, colorize: bool = True) -> str:- # pylint: disable=no-name-in-module- if not colorize or not sys.stdout.isatty() or not _tty_supports_color():- return text- try:- from pygments import highlight # noqa: PLC0415- except ImportError:- return text- from pygments.formatters import TerminalFormatter # noqa: PLC0415- from pygments.lexers import PythonLexer # noqa: PLC0415-- return highlight(text, PythonLexer(), TerminalFormatter())---def pformat(obj: Any, *args: Any, **kwargs: Any) -> str:- return _colorize(pformat_(obj), kwargs.pop("colorize", True))---def pprint(obj: Any, *args: Any, **kwargs: Any) -> None:- print(pformat(obj, *args, **kwargs))+# pragma: no file cover+import warnings++from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._colorize import pformat, pprint++warnings.warn(+ "The scrapy.utils.display module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "pformat",+ "pprint",+]scrapy/utils/ftp.pyindex f08e5303d..5bed01a3a 100644@@ -1,53 +1,16 @@-import posixpath-from contextlib import closing-from ftplib import FTP, FTP_TLS, error_perm-from posixpath import dirname-from ssl import create_default_context-from typing import IO+# pragma: no file cover+import warnings +from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._ftp import ftp_makedirs_cwd, ftp_store_file -def ftp_makedirs_cwd(ftp: FTP, path: str, first_call: bool = True) -> None:- """Set the current directory of the FTP connection given in the ``ftp``- argument (as a ftplib.FTP object), creating all parent directories if they- don't exist. The ftplib.FTP object must be already connected and logged in.- """- try:- ftp.cwd(path)- except error_perm:- ftp_makedirs_cwd(ftp, dirname(path), False)- ftp.mkd(path)- if first_call:- ftp.cwd(path)+warnings.warn(+ "The scrapy.utils.ftp module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+) --def ftp_store_file(- *,- path: str,- file: IO[bytes],- host: str,- port: int,- username: str,- password: str,- use_active_mode: bool = False,- overwrite: bool = True,- tls: bool = False,-) -> None:- """Opens a FTP connection with passed credentials, sets current directory- to the directory extracted from given path, then uploads the file to server.-- If *tls* is ``True``, the connection is secured with TLS (FTPS), and the- certificate of the server is verified.- """- ftp = FTP_TLS(context=create_default_context()) if tls else FTP()- with ftp, closing(file):- ftp.connect(host, port)- ftp.login(username, password)- if isinstance(ftp, FTP_TLS):- ftp.prot_p()- if use_active_mode:- ftp.set_pasv(False)- file.seek(0)- dirname, filename = posixpath.split(path)- ftp_makedirs_cwd(ftp, dirname)- command = "STOR" if overwrite else "APPE"- ftp.storbinary(f"{command} {filename}", file)+__all__ = [+ "ftp_makedirs_cwd",+ "ftp_store_file",+]scrapy/utils/log.pyindex bb4254f0c..1d56ff5e1 100644@@ -16,8 +16,8 @@ from twisted.python.failure import Failure import scrapy from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.settings import Settings+from scrapy.utils._colorize import tty_supports_color from scrapy.utils._versions import get_versions-from scrapy.utils.display import _tty_supports_color if TYPE_CHECKING: from types import TracebackType@@ -187,7 +187,7 @@ def _get_formatter(handler: logging.Handler, settings: Settings) -> logging.Form and not isinstance(handler, logging.FileHandler) and settings.getbool("LOG_COLOR") and handler.stream.isatty()- and _tty_supports_color()+ and tty_supports_color() ): try: from colorlog import ColoredFormatter # noqa: PLC0415scrapy/utils/ossignal.pyindex 4ad8e5741..bd9048bd6 100644@@ -1,33 +1,21 @@-from __future__ import annotations+# pragma: no file cover+import warnings -import signal-from collections.abc import Callable-from types import FrameType-from typing import Any, TypeAlias--# copy of _HANDLER from typeshed/stdlib/signal.pyi-SignalHandlerT: TypeAlias = (- Callable[[int, FrameType | None], Any] | int | signal.Handlers | None+from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._ossignals import (+ SignalHandlerT,+ install_shutdown_handlers,+ signal_names, ) -signal_names: dict[int, str] = {member.value: member.name for member in signal.Signals}-+warnings.warn(+ "The scrapy.utils.ossignal module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+) -def install_shutdown_handlers(- function: SignalHandlerT, override_sigint: bool = True-) -> None:- """Install the given function as a signal handler for all common shutdown- signals (such as SIGINT, SIGTERM, etc). If ``override_sigint`` is ``False`` the- SIGINT handler won't be installed if there is already a handler in place- (e.g. Pdb)- """- signal.signal(signal.SIGTERM, function)- if (- signal.getsignal(signal.SIGINT) # pylint: disable=comparison-with-callable- == signal.default_int_handler- or override_sigint- ):- signal.signal(signal.SIGINT, function)- # Catch Ctrl-Break in windows- if hasattr(signal, "SIGBREAK"):- signal.signal(signal.SIGBREAK, function)+__all__ = [+ "SignalHandlerT",+ "install_shutdown_handlers",+ "signal_names",+]scrapy/utils/sitemap.pyindex 03b7bf3b1..cc66f5d1b 100644@@ -1,136 +1,16 @@-"""-Module for processing Sitemaps.--Note: The main purpose of this module is to provide support for the-SitemapSpider, its API is subject to change without notice.-"""--from __future__ import annotations-+# pragma: no file cover import warnings-from io import BytesIO, StringIO-from typing import TYPE_CHECKING, Any-from urllib.parse import urljoin--import lxml.etree from scrapy.exceptions import ScrapyDeprecationWarning--if TYPE_CHECKING:- from collections.abc import Iterable, Iterator---class Sitemap:- """Class to parse Sitemap (type=urlset) and Sitemap Index- (type=sitemapindex) files"""-- __slots__ = ("type", "xmliter")-- def __init__(self, xmltext: str | bytes):- if isinstance(xmltext, str):- warnings.warn(- "Passing `str` type as `xmltext` is deprecated, use `bytes`",- ScrapyDeprecationWarning,- stacklevel=2,- )- xmltext = xmltext.encode()-- self.xmliter = lxml.etree.iterparse(- BytesIO(xmltext),- recover=True,- remove_comments=True,- resolve_entities=False,- remove_blank_text=True,- collect_ids=False,- remove_pis=True,- events=("start", "end"),- )- _, root = next(self.xmliter)- self.type = self._get_tag_name(root)-- def __iter__(self) -> Iterator[dict[str, Any]]:- for event, elem in self.xmliter:- if event == "start":- continue-- if self._get_tag_name(elem) not in {"url", "sitemap"}:- continue-- if d := self._process_sitemap_element(elem):- yield d-- def _process_sitemap_element(- self, elem: lxml.etree._Element- ) -> dict[str, Any] | None:- d: dict[str, Any] = {}- alternate: list[str] = []- has_loc = False-- for el in elem:- try:- tag_name = self._get_tag_name(el)- if not tag_name:- continue-- if tag_name == "link":- if href := el.get("href"):- alternate.append(href)- else:- d[tag_name] = el.text.strip() if el.text else ""- if not has_loc and tag_name == "loc":- has_loc = True- finally:- el.clear()- elem.clear()- parent = elem.getparent()- if parent is not None:- while elem.getprevious() is not None:- del parent[0]-- if not has_loc:- return None-- if alternate:- d["alternate"] = alternate-- return d-- @staticmethod- def _get_tag_name(elem: lxml.etree._Element) -> str:- tag = elem.tag- if not isinstance(tag, str):- return ""- _, _, localname = tag.partition("}")- return localname or tag---def sitemap_urls_from_robots(- robots_text: str | bytes,- base_url: str | None = None,-) -> Iterable[str]:- if isinstance(robots_text, bytes):- for line in BytesIO(robots_text):- if line.lstrip()[:8].lower() == b"sitemap:":- try:- url = line.partition(b":")[2].strip().decode()- except UnicodeDecodeError:- continue- yield urljoin(base_url or "", url)-- else:- warnings.warn(- "Passing `str` type as `robots_text` is deprecated, use `bytes`",- ScrapyDeprecationWarning,- stacklevel=2,- )- yield from _sitemap_urls_from_robots_str(robots_text, base_url)---def _sitemap_urls_from_robots_str(- robots_text: str,- base_url: str | None = None,-) -> Iterable[str]:- for line in StringIO(robots_text):- if line.lstrip()[:8].lower() == "sitemap:":- url = line.partition(":")[2].strip()- yield urljoin(base_url or "", url)+from scrapy.utils._sitemap import Sitemap, sitemap_urls_from_robots++warnings.warn(+ "The scrapy.utils.sitemap module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "Sitemap",+ "sitemap_urls_from_robots",+]scrapy/utils/template.pyindex 977d5a42a..cccaba9a5 100644@@ -1,41 +1,21 @@-"""Helper functions for working with templates"""--from __future__ import annotations--import re-import string-from pathlib import Path-from typing import TYPE_CHECKING, Any--if TYPE_CHECKING:- from os import PathLike---def render_templatefile(path: str | PathLike[str], **kwargs: Any) -> None:- path_obj = Path(path)- raw = path_obj.read_text("utf8")-- content = string.Template(raw).substitute(**kwargs)-- render_path = path_obj.with_suffix("") if path_obj.suffix == ".tmpl" else path_obj-- if path_obj.suffix == ".tmpl":- path_obj.rename(render_path)-- render_path.write_text(content, "utf8")---CAMELCASE_INVALID_CHARS = re.compile(r"[^a-zA-Z\d]")---def string_camelcase(string: str) -> str:- """Convert a word to its CamelCase version and remove invalid chars-- >>> string_camelcase('lost-pound')- 'LostPound'-- >>> string_camelcase('missing_images')- 'MissingImages'-- """- return CAMELCASE_INVALID_CHARS.sub("", string.title())+# pragma: no file cover+import warnings++from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.utils._template import (+ CAMELCASE_INVALID_CHARS,+ render_templatefile,+ string_camelcase,+)++warnings.warn(+ "The scrapy.utils.template module is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+)++__all__ = [+ "CAMELCASE_INVALID_CHARS",+ "render_templatefile",+ "string_camelcase",+]tests/ignores.txtindex 75ae2d141..6893acebc 100644@@ -4,7 +4,12 @@ scrapy/interfaces.py scrapy/mail.py scrapy/utils/benchserver.py scrapy/utils/console.py+scrapy/utils/display.py+scrapy/utils/ftp.py scrapy/utils/gz.py scrapy/utils/job.py+scrapy/utils/ossignal.py+scrapy/utils/sitemap.py scrapy/utils/ssl.py+scrapy/utils/template.py scrapy/utils/versions.pytests/test_utils_display.pyindex b8d15bcba..9a545f9c2 100644@@ -8,7 +8,7 @@ from unittest import mock import pytest -from scrapy.utils.display import _enable_windows_terminal_processing, pformat, pprint+from scrapy.utils._colorize import _enable_windows_terminal_processing, pformat, pprint if TYPE_CHECKING: from collections.abc import Mapping, Sequence@@ -59,7 +59,7 @@ def test_pformat_old_windows(isatty: mock.Mock, version: mock.Mock) -> None: @mock.patch("sys.platform", "win32")-@mock.patch("scrapy.utils.display._enable_windows_terminal_processing")+@mock.patch("scrapy.utils._colorize._enable_windows_terminal_processing") @mock.patch("platform.version") @mock.patch("sys.stdout.isatty") def test_pformat_windows_no_terminal_processing(@@ -72,7 +72,7 @@ def test_pformat_windows_no_terminal_processing( @mock.patch("sys.platform", "win32")-@mock.patch("scrapy.utils.display._enable_windows_terminal_processing")+@mock.patch("scrapy.utils._colorize._enable_windows_terminal_processing") @mock.patch("platform.version") @mock.patch("sys.stdout.isatty") def test_pformat_windows(tests/test_utils_log.pyindex ce8da0b39..825d281d9 100644@@ -163,7 +163,7 @@ class TestGetFormatter: self, monkeypatch: pytest.MonkeyPatch ) -> None: pytest.importorskip("colorlog")- monkeypatch.setattr("scrapy.utils.log._tty_supports_color", lambda: True)+ monkeypatch.setattr("scrapy.utils.log.tty_supports_color", lambda: True) stream = _TTYStringIO() handler = logging.StreamHandler(stream) handler.setFormatter(_get_formatter(handler, self._settings()))@@ -186,7 +186,7 @@ class TestGetFormatter: def test_plain_when_log_color_disabled( self, monkeypatch: pytest.MonkeyPatch ) -> None:- monkeypatch.setattr("scrapy.utils.log._tty_supports_color", lambda: True)+ monkeypatch.setattr("scrapy.utils.log.tty_supports_color", lambda: True) handler = logging.StreamHandler(_TTYStringIO()) formatter = _get_formatter(handler, self._settings(LOG_COLOR=False)) assert type(formatter) is logging.Formatter@@ -202,7 +202,7 @@ class TestGetFormatter: def test_plain_when_colorlog_not_installed( self, monkeypatch: pytest.MonkeyPatch ) -> None:- monkeypatch.setattr("scrapy.utils.log._tty_supports_color", lambda: True)+ monkeypatch.setattr("scrapy.utils.log.tty_supports_color", lambda: True) monkeypatch.setitem(sys.modules, "colorlog", None) handler = logging.StreamHandler(_TTYStringIO()) formatter = _get_formatter(handler, self._settings())tests/test_utils_ossignal.pyindex 4c8f7d0b4..e38a853b6 100644@@ -5,7 +5,7 @@ from typing import TYPE_CHECKING, Any import pytest -from scrapy.utils.ossignal import install_shutdown_handlers+from scrapy.utils._ossignals import install_shutdown_handlers if TYPE_CHECKING: from collections.abc import Generatortests/test_utils_sitemap.pyindex dda13006d..571ad4075 100644@@ -3,7 +3,7 @@ from __future__ import annotations import pytest from scrapy.exceptions import ScrapyDeprecationWarning-from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots+from scrapy.utils._sitemap import Sitemap, sitemap_urls_from_robots def test_sitemap():tests/test_utils_template.pyindex 076edf3db..fed314974 100644@@ -2,7 +2,7 @@ from __future__ import annotations from typing import TYPE_CHECKING -from scrapy.utils.template import render_templatefile+from scrapy.utils._template import render_templatefile if TYPE_CHECKING: from pathlib import Path