Backtestsscrapyrun-20261025-211112#5050

Issue #5050 Merged fix PR #7988 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

H2DownloadHandler: make acceptable server frame size configurable

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches add a setting that is advertised to the server and both pass all seven independent tests, so the core ask is met by either. The differences are in robustness, edge handling and tests.

Correctness: Patch A builds the setting into the initial SETTINGS frame and also sets `conn.max_inbound_frame_size` directly, so larger frames are accepted immediately, independent of whether the server ever ACKs the client's settings. That matters for the issue's second use case (servers that ignore the advertised maximum, which are also the servers least likely to behave well). Patch B sends a second SETTINGS frame via `update_settings`; h2 only raises `max_inbound_frame_size` when the server ACKs, so acceptance of large frames depends on ACK ordering. In normal flows the ACK precedes response data, so B works in practice (the independent tests confirm), but it is strictly less robust. The issue explicitly names the standard's limits; A validates the range with a clear ValueError at handler init and in the protocol constructor, while B does no validation, so an out-of-range value surfaces as an h2 `InvalidSettingsValueError` thrown from `connectionMade` on every connection, which is a poor failure mode (logged by Twisted, requests likely hang). A's replacement of `conn.local_settings` reaches into h2 state more than B's use of the public `update_settings` API, but `local_settings`/`max_inbound_frame_size` are public attributes, A preserves the other defaults (and tests that), and the comment explains why.

Tests: A adds a focused in-memory h2 server suite that covers the default being advertised, a custom value being advertised with other settings preserved, rejection above the default, acceptance when configured, rejection above a custom value, both boundary values, and invalid values at both the protocol and handler level, including the exact 'server ignores the advertised maximum' scenario by forcing the server's outbound frame size. B adds one parametrized end-to-end test against the Twisted server; it would fail on base only on the final `local_settings.max_frame_size` assertion, never asserts that frames larger than 16384 were actually sent or received, and does not cover rejection, boundaries or invalid values.

Scope/craft: Both remove the known-limitation note and document the setting. B's docs add a useful caveat about DOWNLOAD_MAXSIZE/WARNSIZE being per-frame and note that the httpx handler ignores the setting, and `HTTP2_MAX_FRAME_SIZE` arguably fits settings naming slightly better than `H2_MAX_FRAME_SIZE`; these are minor pluses for B. A's double validation (handler and protocol) is slight duplication but intentional fail-fast. On balance A resolves the issue more completely, handles the standard's limits the issue calls out, and backs it with far stronger tests.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

7m 24s

Wall-clock time in the container

Full test suite

No new failures

126 passed · 200 skipped · 326 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

208vs 56 merged

Changed lines, ours vs merged

The current implementation of HTTP/2 support in Scrapy requires servers to limit the frame size of the data that they send to the lowest maximum size that the standard defines, which is 16 384 bytes.

We should make this maximum configurable, at least within the limits that the standard defines (16 384 - 16 777 215), so that:

  • The value can be fine-tuned for performance reasons.
  • Scrapy can work with servers that ignore the maximum frame size indicated by Scrapy and send larger frames, which Scrapy does not allow (and should probably not allow, for security reasons).

The two patches

Our patch+204 −4
docs/topics/download-handlers.rstindex d95e7b52b..4ccc41d7f 100644@@ -209,15 +209,35 @@ Known limitations of the HTTP/2 support: -   No support for HTTP/2 Cleartext (h2c), since no major browser supports     HTTP/2 unencrypted (refer `http2 faq`_). --   No setting to specify a maximum `frame size`_ larger than the default-    value, 16384. Connections to servers that send a larger frame will fail.- -   No support for `server pushes`_, which are ignored. -.. _frame size: https://datatracker.ietf.org/doc/html/rfc7540#section-4.2 .. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption .. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2 +.. setting:: H2_MAX_FRAME_SIZE++H2_MAX_FRAME_SIZE+^^^^^^^^^^^^^^^^^++.. versionadded:: VERSION++Default: ``16384`` (2\ :sup:`14`)++Maximum `frame size`_, in bytes, that this handler accepts from servers.++The value is sent to servers as ``SETTINGS_MAX_FRAME_SIZE``, and any frame+larger than it is treated as a connection error: the connection is aborted and+all requests sent over it fail.++It must be between ``16384`` (2\ :sup:`14`) and ``16777215``+(2\ :sup:`24` - 1), both included, as defined by the HTTP/2 standard;+otherwise a :exc:`ValueError` is raised when the handler is initialized.++Increase it to fine-tune performance, or to work with servers that ignore the+advertised maximum and send larger frames.++.. _frame size: https://datatracker.ietf.org/doc/html/rfc7540#section-4.2+ HTTP11DownloadHandler --------------------- scrapy/core/_http2/protocol.pyindex aff985d94..d5cd45103 100644@@ -21,6 +21,7 @@ from h2.events import (     WindowUpdated, ) from h2.exceptions import FrameTooLargeError, H2Error+from h2.settings import SettingCodes, Settings from twisted.internet.interfaces import (     IAddress,     IHandshakeListener,@@ -45,6 +46,7 @@ if TYPE_CHECKING:     from twisted.web.client import URI      from scrapy.crawler import Crawler+    from scrapy.settings import BaseSettings     from scrapy.spiders import Spider  @@ -53,6 +55,22 @@ logger = logging.getLogger(__name__)  PROTOCOL_NAME = b"h2" +# Allowed range for SETTINGS_MAX_FRAME_SIZE (RFC 7540, Section 6.5.2).+MIN_MAX_FRAME_SIZE = 2**14+MAX_MAX_FRAME_SIZE = 2**24 - 1+++def _get_max_frame_size(settings: BaseSettings) -> int:+    """Return the validated value of the ``H2_MAX_FRAME_SIZE`` setting."""+    max_frame_size = settings.getint("H2_MAX_FRAME_SIZE")+    if not MIN_MAX_FRAME_SIZE <= max_frame_size <= MAX_MAX_FRAME_SIZE:+        raise ValueError(+            f"Invalid H2_MAX_FRAME_SIZE value: {max_frame_size}. It must be "+            f"between {MIN_MAX_FRAME_SIZE} and {MAX_MAX_FRAME_SIZE} (both "+            f"included)."+        )+    return max_frame_size+  class InvalidNegotiatedProtocol(H2Error):     def __init__(self, negotiated_protocol: bytes) -> None:@@ -112,6 +130,22 @@ class H2ClientProtocol(Protocol, TimeoutMixin):         config = H2Configuration(client_side=True, header_encoding="utf-8")         self.conn = H2Connection(config=config) +        # Advertise the configured maximum frame size in the initial SETTINGS+        # frame sent by initiate_connection(), and accept frames of up to that+        # size right away instead of only after the remote acknowledges it.+        max_frame_size = _get_max_frame_size(crawler.settings)+        self.conn.local_settings = Settings(+            client=True,+            initial_values={+                **{+                    SettingCodes(code): value+                    for code, value in self.conn.local_settings.items()+                },+                SettingCodes.MAX_FRAME_SIZE: max_frame_size,+            },+        )+        self.conn.max_inbound_frame_size = max_frame_size+         # ID of the next request stream         # Following the convention - 'Streams initiated by a client MUST         # use odd-numbered stream identifiers' (RFC 7540 - Section 5.1.1)scrapy/core/downloader/handlers/http2.pyindex 935a2ae4c..e4cc8bf5c 100644@@ -5,6 +5,7 @@ from typing import TYPE_CHECKING from urllib.parse import urldefrag  from scrapy.core._http2.agent import H2Agent, H2ConnectionPool+from scrapy.core._http2.protocol import _get_max_frame_size from scrapy.core.downloader.contextfactory import _load_context_factory_from_settings from scrapy.core.downloader.handlers.base import BaseDownloadHandler from scrapy.exceptions import (@@ -37,6 +38,8 @@ class H2DownloadHandler(BaseDownloadHandler):             raise NotConfigured(f"{type(self).__name__} requires a Twisted reactor.")         super().__init__(crawler)         self._crawler = crawler+        # Fail early on an invalid value instead of on every new connection.+        _get_max_frame_size(crawler.settings)          from twisted.internet import reactor scrapy/settings/default_settings.pyindex d7b30b849..d99b4a98f 100644@@ -108,6 +108,7 @@ __all__ = [     "FTP_PASSWORD",     "FTP_USER",     "GCS_PROJECT_ID",+    "H2_MAX_FRAME_SIZE",     "HTTPAUTH_DOMAIN",     "HTTPAUTH_PASS",     "HTTPAUTH_USER",@@ -402,6 +403,8 @@ FTP_PASSWORD = "guest"  # noqa: S105  GCS_PROJECT_ID = None +H2_MAX_FRAME_SIZE = 2**14  # 16 KiB, the HTTP/2 default+ HTTPAUTH_USER = "" HTTPAUTH_PASS = "" HTTPAUTH_DOMAIN = Nonetests/test_http2_client_protocol.pyindex 161015231..e0ba3b643 100644@@ -12,9 +12,11 @@ from urllib.parse import urlencode  import pytest from pytest_twisted import async_yield_fixture+from twisted.internet.address import IPv4Address as TxIPv4Address from twisted.internet.defer import Deferred, DeferredList, inlineCallbacks from twisted.internet.endpoints import SSL4ClientEndpoint, SSL4ServerEndpoint from twisted.internet.ssl import Certificate, PrivateCertificate, optionsForClientTLS+from twisted.internet.testing import StringTransport from twisted.web.client import URI, ResponseFailed from twisted.web.http import H2_ENABLED from twisted.web.http import Request as TxRequest@@ -36,6 +38,8 @@ from tests.mockserver.utils import ssl_context_factory if TYPE_CHECKING:     from collections.abc import AsyncGenerator, Callable, Coroutine, Generator +    from h2.connection import H2Connection+     from scrapy.core._http2.protocol import H2ClientProtocol  @@ -765,3 +769,139 @@ class TestHttps2ClientProtocol:             k_decoded, v_decoded = str(k, "utf-8"), str(v[0], "utf-8")             assert k_decoded in response_headers             assert v_decoded == response_headers[k_decoded]+++class TestMaxFrameSize:+    """Tests for the H2_MAX_FRAME_SIZE setting, using an in-memory h2 server+    that ignores the maximum frame size advertised by the client."""++    host = "example.com"+    body_size = 2**15  # Larger than the default maximum frame size, 2**14.++    def _connect(+        self, settings: dict[str, Any] | None = None+    ) -> tuple[H2ClientProtocol, StringTransport, H2Connection]:+        from h2.config import H2Configuration  # noqa: PLC0415+        from h2.connection import H2Connection  # noqa: PLC0415++        from scrapy.core._http2.protocol import H2ClientProtocol  # noqa: PLC0415++        uri = URI.fromBytes(f"https://{self.host}/".encode())+        protocol = H2ClientProtocol(+            uri, get_crawler(settings_dict=settings), Deferred()+        )+        transport = StringTransport(+            peerAddress=TxIPv4Address("TCP", "127.0.0.1", 443)+        )+        transport.getPeerCertificate = mock.Mock(return_value=None)+        protocol.makeConnection(transport)++        server = H2Connection(+            config=H2Configuration(client_side=False, header_encoding="utf-8")+        )+        server.initiate_connection()+        self._exchange(protocol, transport, server)+        return protocol, transport, server++    @staticmethod+    def _exchange(+        protocol: H2ClientProtocol, transport: StringTransport, server: H2Connection+    ) -> list[Any]:+        events = server.receive_data(transport.value())+        transport.clear()+        protocol.dataReceived(server.data_to_send())+        return events++    def _respond_with_large_frame(+        self,+        protocol: H2ClientProtocol,+        transport: StringTransport,+        server: H2Connection,+    ) -> Deferred[Response]:+        from h2.events import RequestReceived  # noqa: PLC0415++        d = protocol.request(Request(f"https://{self.host}/"), DummySpider())+        events = self._exchange(protocol, transport, server)+        (stream_id,) = [e.stream_id for e in events if isinstance(e, RequestReceived)]+        server.send_headers(+            stream_id,+            [(":status", "200"), ("content-length", str(self.body_size))],+        )+        # Ignore the maximum frame size advertised by the client.+        server.max_outbound_frame_size = self.body_size+        server.send_data(stream_id, b"a" * self.body_size, end_stream=True)+        protocol.dataReceived(server.data_to_send())+        return d++    def test_default_advertised(self):+        protocol, _, server = self._connect()+        assert protocol.conn.max_inbound_frame_size == 2**14+        assert server.remote_settings.max_frame_size == 2**14++    def test_custom_advertised(self):+        protocol, _, server = self._connect({"H2_MAX_FRAME_SIZE": 2**20})+        assert protocol.conn.max_inbound_frame_size == 2**20+        assert server.remote_settings.max_frame_size == 2**20+        # Other default local settings are preserved.+        assert server.remote_settings.max_concurrent_streams == 100++    @staticmethod+    def _assert_frame_too_large(+        protocol: H2ClientProtocol, transport: StringTransport, d: Deferred[Response]+    ) -> None:+        from h2.exceptions import FrameTooLargeError  # noqa: PLC0415++        assert transport.disconnecting+        assert any(+            isinstance(e, FrameTooLargeError) for e in protocol._conn_lost_errors+        )+        assert not d.called++    def test_frame_larger_than_default_rejected(self):+        protocol, transport, server = self._connect()+        d = self._respond_with_large_frame(protocol, transport, server)+        self._assert_frame_too_large(protocol, transport, d)++    def test_frame_larger_than_default_accepted(self):+        protocol, transport, server = self._connect(+            {"H2_MAX_FRAME_SIZE": self.body_size}+        )+        d = self._respond_with_large_frame(protocol, transport, server)+        assert not transport.disconnecting+        results: list[Response] = []+        d.addCallback(results.append)+        (response,) = results+        assert response.status == 200+        assert response.body == b"a" * self.body_size++    def test_frame_larger_than_custom_rejected(self):+        protocol, transport, server = self._connect(+            {"H2_MAX_FRAME_SIZE": self.body_size - 1}+        )+        d = self._respond_with_large_frame(protocol, transport, server)+        self._assert_frame_too_large(protocol, transport, d)++    @pytest.mark.parametrize("value", [2**14 - 1, 2**24, 0, -1])+    def test_invalid_value_protocol(self, value):+        from scrapy.core._http2.protocol import H2ClientProtocol  # noqa: PLC0415++        uri = URI.fromBytes(f"https://{self.host}/".encode())+        with pytest.raises(ValueError, match="H2_MAX_FRAME_SIZE"):+            H2ClientProtocol(+                uri, get_crawler(settings_dict={"H2_MAX_FRAME_SIZE": value}), Deferred()+            )++    @pytest.mark.parametrize("value", [2**14 - 1, 2**24])+    def test_invalid_value_handler(self, value):+        from scrapy.core.downloader.handlers.http2 import (  # noqa: PLC0415+            H2DownloadHandler,+        )++        crawler = get_crawler(settings_dict={"H2_MAX_FRAME_SIZE": value})+        with pytest.raises(ValueError, match="H2_MAX_FRAME_SIZE"):+            H2DownloadHandler(crawler)++    @pytest.mark.parametrize("value", [2**14, 2**24 - 1])+    def test_boundary_values(self, value):+        protocol, _, server = self._connect({"H2_MAX_FRAME_SIZE": value})+        assert server.remote_settings.max_frame_size == value 
Merged fix · PR #7988+50 −6
docs/topics/download-handlers.rstindex d95e7b52b..2d04b621c 100644@@ -209,12 +209,8 @@ Known limitations of the HTTP/2 support: -   No support for HTTP/2 Cleartext (h2c), since no major browser supports     HTTP/2 unencrypted (refer `http2 faq`_). --   No setting to specify a maximum `frame size`_ larger than the default-    value, 16384. Connections to servers that send a larger frame will fail.- -   No support for `server pushes`_, which are ignored. -.. _frame size: https://datatracker.ietf.org/doc/html/rfc7540#section-4.2 .. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption .. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2 docs/topics/settings.rstindex 010c9d1c6..e58815fad 100644@@ -1499,6 +1499,28 @@ Default: ``None``  The Project ID that will be used when storing data on `Google Cloud Storage`_. +.. setting:: HTTP2_MAX_FRAME_SIZE++HTTP2_MAX_FRAME_SIZE+--------------------++.. versionadded:: VERSION++Default: ``16384``++Maximum `frame size`_, in bytes, that servers may send, between ``16384`` and+``16777215``. Connections to servers that send a larger frame fail.++Raise it for servers that send larger frames regardless of this value. Note+that :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` are checked+once per received frame, so a higher value allows a response to exceed them by+more before being caught.++:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` ignores+this setting, as ``httpx`` does not allow configuring the frame size.++.. _frame size: https://datatracker.ietf.org/doc/html/rfc7540#section-4.2+ .. setting:: ITEM_PIPELINES  ITEM_PIPELINESscrapy/core/_http2/protocol.pyindex aff985d94..1e673ce8b 100644@@ -21,6 +21,7 @@ from h2.events import (     WindowUpdated, ) from h2.exceptions import FrameTooLargeError, H2Error+from h2.settings import SettingCodes from twisted.internet.interfaces import (     IAddress,     IHandshakeListener,@@ -261,6 +262,9 @@ class H2ClientProtocol(Protocol, TimeoutMixin):          # Initiate H2 Connection         self.conn.initiate_connection()+        max_frame_size = self._crawler.settings.getint("HTTP2_MAX_FRAME_SIZE")+        if max_frame_size != self.conn.local_settings.max_frame_size:+            self.conn.update_settings({SettingCodes.MAX_FRAME_SIZE: max_frame_size})         self._write_to_transport()      def _lose_connection_with_error(self, errors: list[BaseException]) -> None:scrapy/settings/default_settings.pyindex d7b30b849..b97844103 100644@@ -108,6 +108,7 @@ __all__ = [     "FTP_PASSWORD",     "FTP_USER",     "GCS_PROJECT_ID",+    "HTTP2_MAX_FRAME_SIZE",     "HTTPAUTH_DOMAIN",     "HTTPAUTH_PASS",     "HTTPAUTH_USER",@@ -402,6 +403,8 @@ FTP_PASSWORD = "guest"  # noqa: S105  GCS_PROJECT_ID = None +HTTP2_MAX_FRAME_SIZE = 16384+ HTTPAUTH_USER = "" HTTPAUTH_PASS = "" HTTPAUTH_DOMAIN = Nonetests/test_http2_client_protocol.pyindex 161015231..075f1acb9 100644@@ -37,6 +37,7 @@ if TYPE_CHECKING:     from collections.abc import AsyncGenerator, Callable, Coroutine, Generator      from scrapy.core._http2.protocol import H2ClientProtocol+    from scrapy.crawler import Crawler   pytestmark = [@@ -236,9 +237,16 @@ class TestHttps2ClientProtocol:         ) + self.certificate_file.read_text(encoding="utf-8")         return PrivateCertificate.loadPEM(pem)  # type: ignore[no-any-return] +    @pytest.fixture+    def crawler(self, request: pytest.FixtureRequest) -> Crawler:+        return get_crawler(settings_dict=getattr(request, "param", None))+     @async_yield_fixture  # type: ignore[untyped-decorator]     async def client(-        self, server_port: int, client_certificate: PrivateCertificate+        self,+        server_port: int,+        client_certificate: PrivateCertificate,+        crawler: Crawler,     ) -> AsyncGenerator[H2ClientProtocol]:         from twisted.internet import reactor @@ -250,7 +258,7 @@ class TestHttps2ClientProtocol:             acceptableProtocols=[b"h2"],         )         uri = URI.fromBytes(bytes(self.get_url(server_port, "/"), "utf-8"))-        h2_client_factory = H2ClientFactory(uri, get_crawler(), Deferred())+        h2_client_factory = H2ClientFactory(uri, crawler, Deferred())         client_endpoint = SSL4ClientEndpoint(             reactor, self.host, server_port, client_options         )@@ -312,6 +320,17 @@ class TestHttps2ClientProtocol:         request = Request(self.get_url(server_port, "/get-data-html-large"))         await self._check_GET(client, request, Data.HTML_LARGE, 200) +    @pytest.mark.parametrize(+        "crawler", [{"HTTP2_MAX_FRAME_SIZE": 1024**2}], indirect=True+    )+    @deferred_f_from_coro_f+    async def test_GET_large_frames(+        self, server_port: int, client: H2ClientProtocol+    ) -> None:+        request = Request(self.get_url(server_port, "/get-data-html-large"))+        await self._check_GET(client, request, Data.HTML_LARGE, 200)+        assert client.conn.local_settings.max_frame_size == 1024**2+     async def _check_GET_x10(         self,         client: H2ClientProtocol,