Backtestsscrapyrun-20261025-211112#5906
Add possibility to use Selector (bytes) added in parsel 1.8.0.
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Patch B does not actually achieve the issue's goal. It passes body/encoding to parsel, but the very next line still calls get_base_url(response), which on base reads response.text[0:4096] and thereby decodes and caches the entire body as a str. The independent tests confirm this: every 'does not use Response.text' and memory-peak test fails on B exactly as on base ('Response.text was used', retained decoded copy of the same size). So B's change is effectively a no-op for RAM. B also has correctness gaps: it feeds response.encoding directly to parsel, ignoring that TextResponse.text gives a BOM precedence over declared encodings, and it ignores the gb18030 custom error handler and utf-16/32 endianness defaults, so output can diverge from the previous .text-based behaviour. Beyond that, B bumps parsel/cssselect minimums and strips jmespath compat code and tests, which is unrelated churn with compatibility risk, and it adds no test that would fail before and pass after. Patch A addresses the root cause end to end: it builds the selector from bytes (gated on a parsel version that decodes in Python rather than truncating at bad bytes), and it adds _text_head so get_base_url/get_meta_refresh decode only the first few KB instead of the whole body, which is what makes the memory saving real. It mirrors TextResponse.text semantics (BOM precedence, utf-16/32 BE default, gb18030 and unknown-encoding fallbacks, cached-ubody reuse), keeps the root-ignored warning path, and ships substantive tests (forbid-text mocks, equivalence-with-.text parametrizations, head-decoding boundary cases, old-parsel fallback). Its skipif only triggers on old parsel, which is the fallback path by design. All independent tests pass on A and the encoding-correctness tests remain green. A is clearly the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
15m 43s
Wall-clock time in the container
Full test suite
No new failures
961 passed · 31 skipped · 992 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
280vs 58 merged
Changed lines, ours vs merged
The issue
View on GitHubSummary
Please review possibility to apply RAM memory efficient Selector that can accept bytes as input
added as result of https://github.com/scrapy/parsel/issues/210 (added in parsel 1.8.0) in scrapy (as default)
Currently UnifiedSelector (subclass of parsel's Selector used in scrapy) configured to use Response.text as input
Response.text (property) -> creates new object (Response.body -> converted to str) which is memory intensive and not needed
https://github.com/scrapy/scrapy/blob/5a37af146f4f036168ac562918fca43adb4ac65f/scrapy/selector/unified.py#L66-L73
Motivation
As mentioned on https://github.com/scrapy/parsel/issues/210 usage of str input for creating Selector object is more RAM memory intensive
Describe alternatives you've considered
At this stage it will require to.. separately create Selector object with bytes input inside spiders callback method (and not use Response.Selector)
Additional context
Removing other usages of Response.text will significantly reduce RAM required to processing response
The two patches
scrapy/selector/unified.pyindex 1c90c75cb..9e448d61f 100644@@ -1,10 +1,13 @@ from __future__ import annotations +import codecs from typing import Any, Literal from parsel import Selector as _ParselSelector+from w3lib.encoding import read_bom, resolve_encoding from scrapy.http import HtmlResponse, JsonResponse, TextResponse, XmlResponse+from scrapy.utils import _deps_compat from scrapy.utils.python import to_bytes from scrapy.utils.response import get_base_url from scrapy.utils.trackref import object_ref@@ -17,6 +20,29 @@ _NOT_SET = object() SelectorType = Literal["html", "xml", "json", "text"] +def _body_kwargs(response: TextResponse) -> dict[str, Any] | None:+ """Return the parsel keyword arguments to build a selector from the body+ of *response*, or ``None`` if :attr:`TextResponse.text` must be used."""+ body = response.body+ # Building from bytes, which needs parsel 1.8.0+, avoids keeping a decoded+ # copy of the body in memory. parsel ignores an empty body.+ if not _deps_compat.PARSEL_SUPPORTS_BODY or not body:+ return None+ # TextResponse.text gives a BOM precedence over any other encoding source.+ encoding = resolve_encoding(read_bom(body)[0] or response.encoding)+ # The decoded body exists already if the encoding was inferred, and it may+ # use an encoding name that is not resolved, e.g. ascii.+ if encoding is None or response._cached_ubody is not None:+ return None+ # w3lib decodes gb18030 with a custom error handler that parsel lacks.+ if codecs.lookup(encoding).name == "gb18030":+ return None+ # Like w3lib.encoding.html_to_unicode, without a BOM.+ if encoding in {"utf-16", "utf-32"}:+ encoding += "-be"+ return {"body": body, "encoding": encoding}++ def _response_from_text(text: str | bytes, st: SelectorType | None) -> TextResponse: rt: type[TextResponse] = XmlResponse if st == "xml" else HtmlResponse return rt(url="about:blank", encoding="utf-8", body=to_bytes(text, "utf-8"))@@ -100,7 +126,20 @@ class Selector(_ParselSelector, object_ref): response = _response_from_text(text, type) if response is not None:- text = response.text+ # Only the HTML and XML parsing of parsel decodes bytes like+ # TextResponse.text does, e.g. JSON detection from bytes is limited+ # to UTF-8. With root, text keeps the parsel warning about root+ # being ignored.+ body_kwargs = (+ _body_kwargs(response)+ if type in {"html", "xml"} and root is _NOT_SET+ else None+ )+ if body_kwargs is not None:+ text = None+ kwargs.update(body_kwargs)+ else:+ text = response.text kwargs.setdefault("base_url", get_base_url(response)) self.response = responsescrapy/utils/_deps_compat.pyindex 519d83017..7f87c1aeb 100644@@ -20,6 +20,10 @@ TWISTED_TLS_LIMITS_OFFBY1 = TWISTED_VERSION < TxVersion("twisted", 26, 4, 0) PARSEL_VERSION = Version(PARSEL_VERSION_STRING) # Selector.jmespath() support PARSEL_SUPPORTS_JMESPATH = PARSEL_VERSION >= Version("1.8.0")+# Selector(body=..., encoding=...) support that decodes the body in Python when+# lxml cannot, instead of failing on encodings unknown to lxml or truncating+# documents at undecodable bytes, as parsel 1.8.0 does.+PARSEL_SUPPORTS_BODY = PARSEL_VERSION >= Version("1.12.1") PYOPENSSL_VERSION = Version(PYOPENSSL_VERSION_STRING) # SSL.Context.set_cipher_list() creates a temporary connection, making the context immutablescrapy/utils/response.pyindex d06e389b8..b2f05abf9 100644@@ -15,6 +15,7 @@ from weakref import WeakKeyDictionary from twisted.web import http from w3lib import html+from w3lib.encoding import html_to_unicode from scrapy.http.headers import Headers from scrapy.utils.misc import load_object@@ -27,11 +28,39 @@ if TYPE_CHECKING: _baseurl_cache: WeakKeyDictionary[Response, str] = WeakKeyDictionary() +# Characters at the start of a response where base URLs and meta refresh tags+# are looked for.+_HEAD_LENGTH = 4096+# Bytes of a BOM and of _HEAD_LENGTH characters, plus a margin for a cut+# character, in an encoding that uses at most 4 bytes per character.+_HEAD_BYTES = 4 + 4 * (_HEAD_LENGTH + 4)+++def _text_head(response: TextResponse) -> str:+ """Return the first characters of :attr:`TextResponse.text+ <scrapy.http.TextResponse.text>`, decoding only the start of the body."""+ # Accessing the encoding first, like TextResponse.text does.+ charset = f"charset={response.encoding}"+ body = response.body+ # The decoded body exists already if the encoding was inferred, and it may+ # use an encoding name that html_to_unicode resolves differently.+ if response._cached_ubody is not None:+ return response.text[:_HEAD_LENGTH]+ if len(body) <= _HEAD_BYTES:+ return html_to_unicode(charset, body)[1][:_HEAD_LENGTH]+ text = html_to_unicode(charset, body[:_HEAD_BYTES])[1]+ # The replacement characters of a character cut by the end of the decoded+ # bytes, at most 4 bytes, come after the first _HEAD_LENGTH characters,+ # unless an encoding uses more bytes per character.+ if len(text) >= _HEAD_LENGTH + 4:+ return text[:_HEAD_LENGTH]+ return response.text[:_HEAD_LENGTH]+ def get_base_url(response: TextResponse) -> str: """Return the base url of the given response, joined with the response url""" if response not in _baseurl_cache:- text = response.text[0:4096]+ text = _text_head(response) _baseurl_cache[response] = html.get_base_url( text, response.url, response.encoding )@@ -49,7 +78,7 @@ def get_meta_refresh( ) -> tuple[None, None] | tuple[float, str]: """Parse the http-equiv refresh parameter from the given response""" if response not in _metaref_cache:- text = response.text[0:4096]+ text = _text_head(response) _metaref_cache[response] = html.get_meta_refresh( text, get_base_url(response), response.encoding, ignore_tags=ignore_tags )tests/test_selector.pyindex aa43ce249..60d7603f3 100644@@ -1,10 +1,12 @@+import codecs import weakref+from unittest import mock import pytest from scrapy.http import HtmlResponse, JsonResponse, TextResponse, XmlResponse from scrapy.selector import Selector-from scrapy.utils._deps_compat import PARSEL_SUPPORTS_JMESPATH+from scrapy.utils._deps_compat import PARSEL_SUPPORTS_BODY, PARSEL_SUPPORTS_JMESPATH class TestSelector:@@ -128,6 +130,134 @@ class TestSelector: Selector(TextResponse(url="http://example.com", body=b""), text="") +def _forbid_text():+ return mock.patch.object(+ TextResponse,+ "text",+ new_callable=mock.PropertyMock,+ side_effect=AssertionError("TextResponse.text was used"),+ )+++@pytest.mark.skipif(+ not PARSEL_SUPPORTS_BODY, reason="parsel < 1.12.1 doesn't fully support body"+)+class TestSelectorFromBody:+ @pytest.mark.parametrize(+ ("response_cls", "body", "query", "expected"),+ [+ (HtmlResponse, b"<html><p>caf\xc3\xa9</p></html>", "//p/text()", "caf\xe9"),+ (XmlResponse, b"<root><p>caf\xc3\xa9</p></root>", "//p/text()", "caf\xe9"),+ ],+ )+ def test_response_text_not_used(self, response_cls, body, query, expected):+ response = response_cls("http://example.com", body=body, encoding="utf-8")+ with _forbid_text():+ sel = response.selector+ assert sel.xpath(query).get() == expected+ assert response.xpath(query).get() == expected+ assert response.css("p::text").get() == expected+ assert response._cached_ubody is None++ def test_explicit_type_response_text_not_used(self):+ response = TextResponse(+ "http://example.com", body=b"<div><p>Hi</p></div>", encoding="utf-8"+ )+ with _forbid_text():+ assert Selector(response, type="html").css("p::text").get() == "Hi"+ assert Selector(response, type="xml").xpath("//p/text()").get() == "Hi"++ def test_text_argument(self):+ with _forbid_text():+ sel = Selector(text="<p>caf\xe9</p>")+ assert sel.css("p::text").get() == "caf\xe9"+ sel = Selector(text="<root><p>caf\xe9</p></root>", type="xml")+ assert sel.xpath("//p/text()").get() == "caf\xe9"++ def test_empty_body(self):+ for response in (+ HtmlResponse("http://example.com", body=b""),+ XmlResponse("http://example.com", body=b""),+ ):+ assert response.selector.xpath("//text()").getall() == []+ assert Selector(text="").xpath("//text()").getall() == []++ @pytest.mark.parametrize(+ ("body", "kwargs"),+ [+ # declared latin-1 is decoded as cp1252, like TextResponse.text+ (b"<html><p>\x80caf\xe9</p></html>", {"encoding": "latin1"}),+ (+ b'<html><head><meta charset="iso-8859-1"></head>'+ b"<p>\x80caf\xe9</p></html>",+ {},+ ),+ (+ b"<html><p>\x80caf\xe9</p></html>",+ {"headers": {"Content-Type": "text/html; charset=iso-8859-1"}},+ ),+ # undecodable bytes are replaced, like in TextResponse.text+ (b"<html><p>\x80caf\xe9\xff</p></html>", {"encoding": "utf-8"}),+ # a BOM takes precedence over the declared encoding+ (+ codecs.BOM_UTF8 + "<html><p>caf\xe9</p></html>".encode(),+ {"encoding": "latin1"},+ ),+ (+ codecs.BOM_UTF16_LE + "<html><p>caf\xe9</p></html>".encode("utf-16-le"),+ {},+ ),+ (+ codecs.BOM_UTF16_BE + "<html><p>caf\xe9</p></html>".encode("utf-16-be"),+ {},+ ),+ # utf-16 and utf-32 without a BOM are big endian, like in+ # TextResponse.text+ (+ "<html><p>caf\xe9</p></html>".encode("utf-16-be"),+ {"headers": {"Content-Type": "text/html; charset=utf-16"}},+ ),+ ("<html><p>caf\xe9</p></html>".encode("utf-32-be"), {"encoding": "utf-32"}),+ # gb18030 undecodable bytes are replaced like in TextResponse.text+ (b"<html><p>caf\xa8\xa6\x80z</p></html>", {"encoding": "gb18030"}),+ # an inferred default encoding is used like in TextResponse.text+ (b"\x00\x81\xff<html><p>caf\xe9</p></html>", {}),+ # unknown encodings fall back to TextResponse.text+ (b"<html><p>caf\xc3\xa9</p></html>", {"encoding": "unknown-enc"}),+ ],+ )+ def test_same_as_text(self, body, kwargs):+ response = HtmlResponse("http://example.com", body=body, **kwargs)+ expected = Selector(text=response.text).xpath("//p/text()").get()+ actual = Selector(response).xpath("//p/text()").get()+ assert actual == expected+ assert "caf" in actual++ def test_xml_declared_encoding(self):+ body = (+ b'<?xml version="1.0" encoding="iso-8859-1"?><root><p>caf\xe9</p></root>'+ )+ response = XmlResponse("http://example.com", body=body)+ with _forbid_text():+ assert response.xpath("//p/text()").get() == "caf\xe9"++ def test_root_argument_keeps_warning(self):+ response = HtmlResponse("http://example.com", body=b"<p>a</p>")+ root = Selector(text="<p>b</p>").root+ with pytest.warns(UserWarning, match="root is being ignored"):+ sel = Selector(response, type="html", root=root)+ assert sel.css("p::text").get() == "a"++ def test_old_parsel_uses_text(self):+ response = HtmlResponse(+ "http://example.com", body=b"<p>caf\xc3\xa9</p>", encoding="utf-8"+ )+ with mock.patch("scrapy.utils._deps_compat.PARSEL_SUPPORTS_BODY", False):+ sel = Selector(response)+ assert response._cached_ubody is not None+ assert sel.css("p::text").get() == "caf\xe9"++ @pytest.mark.skipif( not PARSEL_SUPPORTS_JMESPATH, reason="parsel < 1.8 doesn't support jmespath" )tests/test_utils_response.pyindex 0ad9bd7be..2a518fe36 100644@@ -1,7 +1,9 @@ from __future__ import annotations from pathlib import Path+import codecs from time import process_time+from unittest import mock from urllib.parse import urlparse import pytest@@ -9,6 +11,7 @@ import pytest from scrapy.http import HtmlResponse, Response, TextResponse from scrapy.utils.python import to_bytes from scrapy.utils.response import (+ _text_head, get_base_url, get_meta_refresh, open_in_browser,@@ -110,6 +113,73 @@ def test_get_base_url(): assert get_base_url(resp2) == "http://www.example.com" +def test_get_base_url_and_meta_refresh_do_not_decode_whole_body():+ head = (+ b'<html><head><base href="http://www.example.com/img/">'+ b'<meta http-equiv="refresh" content="5;url=next"></head><body>'+ )+ resp = HtmlResponse(+ "http://www.example.com",+ body=head + b"<p>caf\xc3\xa9</p>" * 100_000 + b"</body></html>",+ encoding="utf-8",+ )+ with mock.patch.object(+ TextResponse,+ "text",+ new_callable=mock.PropertyMock,+ side_effect=AssertionError("TextResponse.text was used"),+ ):+ assert get_base_url(resp) == "http://www.example.com/img/"+ assert get_meta_refresh(resp) == (5.0, "http://www.example.com/img/next")+ assert resp._cached_ubody is None+++def test_get_base_url_after_multibyte_characters():+ # 4000 characters, but 8000 bytes, before the base tag+ body = ("\xe9" * 4000 + '<base href="http://b.example/">' + "x" * 50_000).encode()+ resp = HtmlResponse("http://www.example.com", body=body, encoding="utf-8")+ assert get_base_url(resp) == "http://b.example/"+ # The tag must start within the first 4096 characters, as before.+ body = ("\xe9" * 4100 + '<base href="http://b.example/">').encode()+ resp = HtmlResponse("http://www.example.com", body=body, encoding="utf-8")+ assert get_base_url(resp) == "http://www.example.com"+++@pytest.mark.parametrize(+ ("body", "kwargs"),+ [+ (b"<p>short</p>", {"encoding": "utf-8"}),+ ("\U0001f600".encode() * 20_000, {"encoding": "utf-8"}),+ # a 4-byte character cut by the end of the decoded bytes+ (b"a" + "\U0001f600".encode() * 20_000, {"encoding": "utf-8"}),+ (b"\x80\xe9\xff" * 20_000, {"encoding": "utf-8"}),+ (b"\x80\xe9" * 20_000, {"encoding": "latin1"}),+ (+ codecs.BOM_UTF8 + "\xe9".encode() * 20_000,+ {"encoding": "latin1"},+ ),+ (+ codecs.BOM_UTF16_LE + "\xe9\U0001f600".encode("utf-16-le") * 20_000,+ {},+ ),+ (+ b'<meta charset="utf-16">' + "\xe9".encode("utf-16-be") * 20_000,+ {},+ ),+ # more than 4 bytes per character+ ("\xe9\u20ac".encode("utf-7") * 20_000, {"encoding": "utf-7"}),+ ("\xe9".encode("gb18030") * 20_000 + b"\x81", {"encoding": "gb18030"}),+ (b"x" * 20_000, {"encoding": "unknown-enc"}),+ # an inferred default encoding+ (b"\x00\x81\xff" + b"caf\xe9" * 5_000, {}),+ ],+)+def test_text_head_same_as_text(body: bytes, kwargs: dict[str, str]) -> None:+ resp = HtmlResponse("http://www.example.com", body=body, **kwargs)+ head = _text_head(resp)+ assert head == resp.text[:4096]++ def test_response_status_message(): assert response_status_message(200) == "200 OK" assert response_status_message(404) == "404 Not Found"
pyproject.tomlindex d7cca1bfd..e538b74ad 100644@@ -11,13 +11,13 @@ dependencies = [ "aiohttp>=3.13.3", "charset-normalizer>=3.4.0", "cryptography>=41.0.5",- "cssselect>=0.9.1",+ "cssselect>=1.2.0", "defusedxml>=0.7.1", "itemadapter>=0.1.0", "itemloaders>=1.0.1", "lxml>=4.6.4", "packaging",- "parsel>=1.5.0",+ "parsel>=1.8.1", "platformdirs>=2.0.0", "protego>=0.1.15", "pyOpenSSL>=24.3.0",scrapy/http/response/text.pyindex c4c48db1c..f6a6c043e 100644@@ -167,10 +167,6 @@ class TextResponse(Response): return self._cached_selector def jmespath(self, query: str, **kwargs: Any) -> SelectorList:- if not hasattr(self.selector, "jmespath"):- raise AttributeError(- "Please install parsel >= 1.8.1 to get jmespath support"- ) return cast("SelectorList", self.selector.jmespath(query, **kwargs)) def xpath(self, query: str, **kwargs: Any) -> SelectorList:scrapy/selector/unified.pyindex 1c90c75cb..5e8242ea6 100644@@ -100,7 +100,20 @@ class Selector(_ParselSelector, object_ref): response = _response_from_text(text, type) if response is not None:- text = response.text+ # parsel only treats bytes and text equivalently for "html" and+ # "xml", and treats an empty body as absent, unlike empty text.+ if type in ("html", "xml") and response.body:+ encoding = response.encoding+ # None means the encoding was declared, not inferred from+ # the body, so the body hasn't been decoded as a side effect.+ if response._cached_ubody is None:+ kwargs["body"] = response.body+ kwargs["encoding"] = encoding+ text = None+ else:+ text = response.text+ else:+ text = response.text kwargs.setdefault("base_url", get_base_url(response)) self.response = responsescrapy/utils/_deps_compat.pyindex 519d83017..e96f0c417 100644@@ -2,7 +2,6 @@ import sys from OpenSSL import __version__ as PYOPENSSL_VERSION_STRING from packaging.version import Version-from parsel import __version__ as PARSEL_VERSION_STRING from twisted import version as TWISTED_VERSION from twisted.python.versions import Version as TxVersion @@ -17,10 +16,6 @@ TWISTED_TLS_NEW_IMPL = TWISTED_VERSION >= TxVersion("twisted", 26, 4, 0) # lowerMaximumSecurityTo off-by-1, https://github.com/twisted/twisted/issues/10232 TWISTED_TLS_LIMITS_OFFBY1 = TWISTED_VERSION < TxVersion("twisted", 26, 4, 0) -PARSEL_VERSION = Version(PARSEL_VERSION_STRING)-# Selector.jmespath() support-PARSEL_SUPPORTS_JMESPATH = PARSEL_VERSION >= Version("1.8.0")- PYOPENSSL_VERSION = Version(PYOPENSSL_VERSION_STRING) # SSL.Context.set_cipher_list() creates a temporary connection, making the context immutable PYOPENSSL_SET_CIPHER_LIST_TMP_CONN = PYOPENSSL_VERSION < Version("25.2.0")tests/test_selector.pyindex aa43ce249..77799ad02 100644@@ -4,7 +4,6 @@ import pytest from scrapy.http import HtmlResponse, JsonResponse, TextResponse, XmlResponse from scrapy.selector import Selector-from scrapy.utils._deps_compat import PARSEL_SUPPORTS_JMESPATH class TestSelector:@@ -56,9 +55,6 @@ class TestSelector: '<div><img src="a.jpg"><p>Hello</p></div>' ] - @pytest.mark.skipif(- not PARSEL_SUPPORTS_JMESPATH, reason="parsel < 1.8 doesn't support json"- ) def test_flavor_detection_json(self) -> None: response = JsonResponse( "http://example.com", body=b'{"a": "b"}', encoding="utf-8"@@ -66,9 +62,6 @@ class TestSelector: assert Selector(response).type == "json" assert response.jmespath("a").get() == "b" - @pytest.mark.skipif(- not PARSEL_SUPPORTS_JMESPATH, reason="parsel < 1.8 doesn't support json"- ) def test_flavor_detection_json_with_html_body(self) -> None: body = b"<div><p>Hello</p></div>" response = JsonResponse("http://example.com", body=body, encoding="utf-8")@@ -128,9 +121,6 @@ class TestSelector: Selector(TextResponse(url="http://example.com", body=b""), text="") -@pytest.mark.skipif(- not PARSEL_SUPPORTS_JMESPATH, reason="parsel < 1.8 doesn't support jmespath"-) class TestJMESPath: def test_json_has_html(self) -> None: """Sometimes the information is returned in a json wrapper"""@@ -265,15 +255,3 @@ class TestJMESPath: assert resp.xpath("//div/content").jmespath("user[*].age.to_string(@)").re( r"(\d+)" ) == ["18", "32", "22", "25"]---@pytest.mark.skipif(PARSEL_SUPPORTS_JMESPATH, reason="parsel >= 1.8 supports jmespath")-def test_jmespath_not_available() -> None:- body = """- {- "website": {"name": "Example"}- }- """- resp = TextResponse(url="http://example.com", body=body, encoding="utf-8")- with pytest.raises(AttributeError):- resp.jmespath("website.name").get()tox.iniindex 4344b9136..409d2c571 100644@@ -147,10 +147,10 @@ deps = brotlicffi==1.2.0.0; implementation_name == "pypy" charset-normalizer==3.4.0 cryptography==41.0.5- cssselect==0.9.1+ cssselect==1.2.0 itemadapter==0.1.0 lxml==4.6.4- parsel==1.5.0+ parsel==1.8.1 platformdirs==2.0.0 pyOpenSSL==24.3.0 queuelib==1.6.1@@ -328,10 +328,10 @@ deps = brotlicffi==1.2.0.0 charset-normalizer==3.4.0 cryptography==44.0.2- cssselect==0.9.1+ cssselect==1.2.0 itemadapter==0.1.0 lxml==5.3.2- parsel==1.5.0+ parsel==1.8.1 pyOpenSSL==24.3.0 queuelib==1.6.1 service_identity==24.2.0