Backtestsscrapyrun-20261025-211112#3407
Can't get partial response body after timeout
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Patch A's change to scrapy/core/downloader/handlers/http11.py adds `if not isinstance(result, Failure)` without importing `Failure`, so every timeout on the default HTTP/1.1 handler now raises NameError. The independent tests confirm this: on Patch A, not only do the partial-response tests still fail, but the default-behaviour test (data loss not allowed, timeout should raise DownloadTimeoutError) regresses from PASS on base to FAIL with NameError. That is a hard breakage of the most common code path. Patch A also skips its own new HTTP test for handlers that don't opt in and only works for aiohttp. Patch B passes all independent tests across http11 (HTTP and HTTPS), aiohttp, the crawl-level scenarios, and the default-behaviour test. Its design is sound: it keeps the reader reference in _ScrapyAgent and returns the buffered body flagged as dataloss from _cb_timeout only when headers were received and data loss is allowed; otherwise it still raises DownloadTimeoutError. It also extends the behaviour to the HTTP/2 stream, uses the broader httpx.TimeoutException and aioftp's timeout exception set, adds tests for both meta and setting paths plus the no-headers case and FTP, and documents the behaviour in settings.rst. B is strictly better on correctness, tests, and craft.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
22m 47s
Wall-clock time in the container
Full test suite
No new failures
789 passed · 276 skipped · 1,075 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
198vs 132 merged
Changed lines, ours vs merged
The issue
View on GitHubI'm working on scraping some pages that hold the HTTP connection open indefinitely, trickling data to the client little by little. I had hoped to use a short timeout to grab the page up to a certain point, but after the timeout is triggered, I can't get at the data that was received. I thought that setting DOWNLOAD_FAIL_ON_DATALOSS=False would allow partial responses to be processed, but I still get twisted.internet.error.TimeoutError: User timeout caused connection failure. If I use errback= on the request, the errback just gets the TimeoutError object and not the response.
Should setting DOWNLOAD_FAIL_ON_DATALOSS=True allow timed out responses to be processed? Or is there another way to deal with partial responses after a timeout?
The two patches
docs/topics/settings.rstindex 3f5c189af..25424a94f 100644@@ -1202,6 +1202,10 @@ Default: ``180`` The amount of time (in secs) that the downloader will wait before timing out. +To get the part of the response received before the timeout instead of a+:exc:`~scrapy.exceptions.DownloadTimeoutError` exception, set+:setting:`DOWNLOAD_FAIL_ON_DATALOSS` to ``False``.+ .. note:: This timeout can be per-request using the :reqmeta:`download_timeout`@@ -1280,6 +1284,11 @@ response was not properly finished. If ``True``, these responses raise a responses are passed through and the flag ``dataloss`` is added to the response, i.e.: ``'dataloss' in response.flags`` is ``True``. +If ``False``, responses that exceed the :setting:`DOWNLOAD_TIMEOUT` after their+headers have been received are also passed through, with the part of the body+received before the timeout and the ``dataloss`` flag, instead of raising a+:exc:`~scrapy.exceptions.DownloadTimeoutError` exception.+ Optionally, this can be set on a per-request basis by using the :reqmeta:`download_fail_on_dataloss` Request.meta key to ``False``. scrapy/core/_http2/stream.pyindex 6f6df1f81..6b93be588 100644@@ -397,12 +397,25 @@ class Stream: ) logger.warning(warning_msg) + def _keeps_partial_body_on_cancel(self, reason: StreamCloseReason) -> bool:+ """Whether a cancelled stream returns the response received so far,+ flagged as ``dataloss``, instead of an empty one."""+ return (+ reason is StreamCloseReason.CANCELLED+ and not self._fail_on_dataloss+ and self._response["status"] is not None+ )+ def reset_stream(self, reason: StreamCloseReason = StreamCloseReason.RESET) -> None: """Close this stream by sending a RST_FRAME to the remote peer""" # The data received so far is the body of the response built for a- # stopped download, otherwise the buffer is cleared early to avoid+ # stopped download or for a cancelled (e.g. timed out) download with+ # data loss allowed, otherwise the buffer is cleared early to avoid # keeping data in memory for a long time- if reason is not StreamCloseReason.STOP_DOWNLOAD:+ if (+ reason is not StreamCloseReason.STOP_DOWNLOAD+ and not self._keeps_partial_body_on_cancel(reason)+ ): self._response["body"].truncate(0) self.metadata["stream_closed_local"] = True@@ -459,6 +472,12 @@ class Stream: self._fire_response_deferred() # Stream was abruptly ended here+ elif self._keeps_partial_body_on_cancel(reason):+ # Client has cancelled the request (e.g. on a download timeout)+ # after receiving the response headers, and data loss is allowed,+ # so the data received so far is returned+ self._fire_response_deferred(flags=["dataloss"])+ elif reason is StreamCloseReason.CANCELLED: # Client has cancelled the request. Remove all the data # received and fire the response deferred with no flags setscrapy/core/downloader/handlers/_aioftp.pyindex e365123fc..24b1ef628 100644@@ -269,6 +269,10 @@ class AioftpDownloadHandler(BaseStreamingDownloadHandler[_AioftpResponse]): def _is_dataloss_exception(exc: Exception) -> bool: return isinstance(exc, aioftp.StatusCodeError) + @staticmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ return isinstance(exc, _TIMEOUT_EXCEPTIONS)+ def _log_tls_info(self, response: _AioftpResponse, request: Request) -> None: if ssl_object := response.control.writer.get_extra_info("ssl_object"): _log_sslobj_debug_info(ssl_object)scrapy/core/downloader/handlers/_aiohttp.pyindex 566575568..3317ffbbe 100644@@ -157,5 +157,9 @@ class AiohttpDownloadHandler(BaseStreamingDownloadHandler[_ClientResponse]): def _is_dataloss_exception(exc: Exception) -> bool: return isinstance(exc, aiohttp.ClientPayloadError) + @staticmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ return isinstance(exc, TimeoutError)+ async def close(self) -> None: await self._session.close()scrapy/core/downloader/handlers/_base_streaming.pyindex 1888e7a26..6a4c78966 100644@@ -135,6 +135,12 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon """Return True if ``exc`` represents dataloss.""" raise NotImplementedError + @staticmethod+ @abstractmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ """Return True if ``exc`` represents a download timeout."""+ raise NotImplementedError+ @staticmethod def _check_deps_installed() -> None: """Raise NotConfigured if the required deps are not installed."""@@ -237,17 +243,22 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon ) ) except Exception as e:- if not self._is_dataloss_exception(e):+ is_timeout = self._is_timeout_exception(e)+ if not is_timeout and not self._is_dataloss_exception(e): raise fail_on_dataloss: bool = request.meta.get( "download_fail_on_dataloss", self._fail_on_dataloss ) if not fail_on_dataloss:+ # A timeout while reading the body loses the rest of it. return make_response( **make_response_base_args, body=self._get_body(response_body, response, request), flags=["dataloss"], )+ if is_timeout:+ # converted into DownloadTimeoutError by _make_request()+ raise if not self._fail_on_dataloss_warned: logger.warning(get_dataloss_msg(request.url)) self._fail_on_dataloss_warned = Truescrapy/core/downloader/handlers/_httpx.pyindex 61c67b87e..c7b116a93 100644@@ -228,6 +228,10 @@ class HttpxDownloadHandler(_Base): exc, httpx.RemoteProtocolError ) and "peer closed connection without sending complete message body" in str(exc) + @staticmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ return isinstance(exc, httpx.TimeoutException)+ def _log_tls_info(self, response: httpx.Response, request: Request) -> None: network_stream: AsyncNetworkStream = response.extensions["network_stream"] extra_ssl_object = network_stream.get_extra_info("ssl_object")scrapy/core/downloader/handlers/http11.pyindex d00836fa7..2fa40f144 100644@@ -458,6 +458,7 @@ class _ScrapyAgent: self._warnsize: int = warnsize self._fail_on_dataloss: bool = fail_on_dataloss self._txresponse: TxResponse | None = None+ self._reader: _ResponseReader | None = None self._crawler: Crawler = crawler self._tls_verbose_logging: bool = tls_verbose_logging @@ -546,6 +547,13 @@ class _ScrapyAgent: if self._txresponse: self._txresponse._transport.stopProducing() + # The response headers and part of the body were received before the+ # timeout, return what was received if data loss is allowed.+ if self._reader is not None and not self._reader._fail_on_dataloss:+ return cast(+ "_T", self._cb_bodydone(self._reader._result(["dataloss"]), url)+ )+ raise DownloadTimeoutError(f"Getting {url} took longer than {timeout} seconds.") def _cb_latency(self, result: _T, request: Request, start_time: float) -> _T:@@ -612,22 +620,22 @@ class _ScrapyAgent: ) d: Deferred[_ResultT] = Deferred(partial(self._cancel, txresponse=txresponse))- txresponse.deliverBody(- _ResponseReader(- finished=d,- txresponse=txresponse,- request=request,- maxsize=maxsize,- warnsize=warnsize,- reached_warnsize=reached_warnsize,- fail_on_dataloss=fail_on_dataloss,- crawler=self._crawler,- tls_verbose_logging=self._tls_verbose_logging,- )+ reader = _ResponseReader(+ finished=d,+ txresponse=txresponse,+ request=request,+ maxsize=maxsize,+ warnsize=warnsize,+ reached_warnsize=reached_warnsize,+ fail_on_dataloss=fail_on_dataloss,+ crawler=self._crawler,+ tls_verbose_logging=self._tls_verbose_logging, )+ txresponse.deliverBody(reader) - # save response for timeouts+ # save response and reader for timeouts self._txresponse = txresponse+ self._reader = reader return d @@ -698,19 +706,22 @@ class _ResponseReader(Protocol): self._crawler: Crawler = crawler self._tls_verbose_logging: bool = tls_verbose_logging + def _result(+ self, flags: list[str] | None = None, stop_download: StopDownload | None = None+ ) -> _ResultT:+ return {+ "txresponse": self._txresponse,+ "body": self._bodybuf.getvalue(),+ "flags": flags,+ "certificate": self._certificate,+ "ip_address": self._ip_address,+ "stop_download": stop_download,+ }+ def _finish_response( self, flags: list[str] | None = None, stop_download: StopDownload | None = None ) -> None:- self._finished.callback(- {- "txresponse": self._txresponse,- "body": self._bodybuf.getvalue(),- "flags": flags,- "certificate": self._certificate,- "ip_address": self._ip_address,- "stop_download": stop_download,- }- )+ self._finished.callback(self._result(flags, stop_download)) def connectionMade(self) -> None: assert self.transportscrapy/core/downloader/handlers/http2.pyindex 5302a7749..087bc3e4d 100644@@ -15,6 +15,7 @@ from scrapy.exceptions import ( ResponseDataLossError, UnsupportedURLSchemeError, )+from scrapy.http import Response from scrapy.utils._download_handlers import ( get_dataloss_msg, normalize_bind_address,@@ -29,7 +30,7 @@ if TYPE_CHECKING: from twisted.web.iweb import IPolicyForHTTPS from scrapy.crawler import Crawler- from scrapy.http import Request, Response+ from scrapy.http import Request from scrapy.spiders import Spider @@ -137,5 +138,10 @@ class _ScrapyH2Agent: timeout_cl.cancel() return response + # The response headers and part of the body were received before the+ # timeout and data loss is allowed, see Stream.close().+ if isinstance(response, Response) and "dataloss" in response.flags:+ return response+ url = urldefrag(request.url)[0] raise DownloadTimeoutError(f"Getting {url} took longer than {timeout} seconds.")tests/test_downloader_handler_aioftp.pyindex bf98920f7..9b62ab5da 100644@@ -232,10 +232,11 @@ async def test_connection_lost() -> None: def _fake_ftp(- retr_reply: bytes, data: bytes = b""+ retr_reply: bytes, data: bytes = b"", *, hang: bool = False ) -> Callable[[asyncio.StreamReader, asyncio.StreamWriter], Any]: """Return a minimal FTP server connection handler that sends *data*, if- any, and then answers RETR with *retr_reply*."""+ any, and then answers RETR with *retr_reply*, or, if *hang* is True,+ stops responding until the client disconnects.""" async def handle( reader: asyncio.StreamReader, writer: asyncio.StreamWriter@@ -267,6 +268,12 @@ def _fake_ftp( if data: writer.write(b"150 Sending\r\n") w.write(data)+ if hang:+ await w.drain()+ await reader.read()+ w.close()+ data_server.close()+ break w.close() data_server.close() writer.write(retr_reply)@@ -315,6 +322,29 @@ async def test_dataloss_allowed() -> None: assert response.flags == ["dataloss"] +_HANGING = _fake_ftp(b"", data=b"partial", hang=True)+++@coroutine_test+async def test_timeout_during_transfer() -> None:+ async with _serve(_HANGING) as url:+ with pytest.raises(DownloadTimeoutError):+ await _download(Request(url, meta={"download_timeout": 0.5}))+++@coroutine_test+async def test_timeout_during_transfer_dataloss_allowed() -> None:+ async with _serve(_HANGING) as url:+ response = await _download(+ Request(+ url,+ meta={"download_timeout": 0.5, "download_fail_on_dataloss": False},+ )+ )+ assert response.body == b"partial"+ assert response.flags == ["dataloss"]++ @coroutine_test async def test_local_filename(ftp_server: MockFTPServer, tmp_path: Path) -> None: local_filename = str(tmp_path / "file.txt").encode()tests/utils/bases/download_handlers_http.pyindex a5be39934..f21ed2809 100644@@ -435,6 +435,48 @@ class TestHttpBase(ABC): with pytest.raises(DownloadTimeoutError): await maybe_deferred_to_future(d) + @coroutine_test+ async def test_timeout_server_hangs_allow_data_loss(+ self, mockserver: MockServer+ ) -> None:+ # the data received before the timeout is returned+ meta = {"download_timeout": 0.5, "download_fail_on_dataloss": False}+ request = Request(+ mockserver.url("/hang-after-headers", is_secure=self.is_secure), meta=meta+ )+ async with self.get_dh() as download_handler:+ response = await download_handler.download_request(request)+ assert response.status == 200+ assert response.flags == ["dataloss"]+ assert response.body == b"some bytes"++ @coroutine_test+ async def test_timeout_server_hangs_allow_data_loss_via_setting(+ self, mockserver: MockServer+ ) -> None:+ meta = {"download_timeout": 0.5}+ request = Request(+ mockserver.url("/hang-after-headers", is_secure=self.is_secure), meta=meta+ )+ async with self.get_dh(+ {"DOWNLOAD_FAIL_ON_DATALOSS": False}+ ) as download_handler:+ response = await download_handler.download_request(request)+ assert response.status == 200+ assert response.flags == ["dataloss"]+ assert response.body == b"some bytes"++ @coroutine_test+ async def test_timeout_nodata_rcvd_allow_data_loss(+ self, mockserver: MockServer+ ) -> None:+ # without response headers there is no partial response to return+ meta = {"download_timeout": 0.5, "download_fail_on_dataloss": False}+ request = Request(mockserver.url("/wait", is_secure=self.is_secure), meta=meta)+ async with self.get_dh() as download_handler:+ with pytest.raises(DownloadTimeoutError):+ await download_handler.download_request(request)+ @pytest.mark.parametrize("send_header", [True, False]) @coroutine_test async def test_host_header(self, send_header: bool, mockserver: MockServer) -> None:
scrapy/core/downloader/handlers/_aioftp.pyindex e365123fc..5adbf660e 100644@@ -269,6 +269,10 @@ class AioftpDownloadHandler(BaseStreamingDownloadHandler[_AioftpResponse]): def _is_dataloss_exception(exc: Exception) -> bool: return isinstance(exc, aioftp.StatusCodeError) + @staticmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ return isinstance(exc, TimeoutError)+ def _log_tls_info(self, response: _AioftpResponse, request: Request) -> None: if ssl_object := response.control.writer.get_extra_info("ssl_object"): _log_sslobj_debug_info(ssl_object)scrapy/core/downloader/handlers/_aiohttp.pyindex 566575568..3317ffbbe 100644@@ -157,5 +157,9 @@ class AiohttpDownloadHandler(BaseStreamingDownloadHandler[_ClientResponse]): def _is_dataloss_exception(exc: Exception) -> bool: return isinstance(exc, aiohttp.ClientPayloadError) + @staticmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ return isinstance(exc, TimeoutError)+ async def close(self) -> None: await self._session.close()scrapy/core/downloader/handlers/_base_streaming.pyindex 1888e7a26..678ab9f22 100644@@ -135,6 +135,13 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon """Return True if ``exc`` represents dataloss.""" raise NotImplementedError + @staticmethod+ @abstractmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ """Return True if ``exc`` represents the download timeout being hit+ while reading the response body."""+ raise NotImplementedError+ @staticmethod def _check_deps_installed() -> None: """Raise NotConfigured if the required deps are not installed."""@@ -237,27 +244,45 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon ) ) except Exception as e:- if not self._is_dataloss_exception(e):- raise- fail_on_dataloss: bool = request.meta.get(- "download_fail_on_dataloss", self._fail_on_dataloss+ return self._handle_body_exception(+ e,+ request,+ make_response_base_args,+ self._get_body(response_body, response, request), )- if not fail_on_dataloss:- return make_response(- **make_response_base_args,- body=self._get_body(response_body, response, request),- flags=["dataloss"],- )- if not self._fail_on_dataloss_warned:- logger.warning(get_dataloss_msg(request.url))- self._fail_on_dataloss_warned = True- raise ResponseDataLossError(str(e)) from e return make_response( **make_response_base_args, body=self._get_body(response_body, response, request), ) + def _handle_body_exception(+ self,+ exc: Exception,+ request: Request,+ make_response_base_args: _BaseResponseArgs,+ body: bytes,+ ) -> Response:+ """Turn an exception raised while reading the response body into a+ partial response, if the request allows for it, or reraise it."""+ is_dataloss = self._is_dataloss_exception(exc)+ if not is_dataloss and not self._is_timeout_exception(exc):+ raise exc+ fail_on_dataloss: bool = request.meta.get(+ "download_fail_on_dataloss", self._fail_on_dataloss+ )+ if not fail_on_dataloss:+ return make_response(+ **make_response_base_args, body=body, flags=["dataloss"]+ )+ if not is_dataloss:+ # let the timeout be reported as usual+ raise exc+ if not self._fail_on_dataloss_warned:+ logger.warning(get_dataloss_msg(request.url))+ self._fail_on_dataloss_warned = True+ raise ResponseDataLossError(str(exc)) from exc+ @staticmethod def _request_headers(request: Request) -> Headers: """Get a prepared copy of the request headers.scrapy/core/downloader/handlers/_httpx.pyindex 61c67b87e..429b32cc5 100644@@ -228,6 +228,10 @@ class HttpxDownloadHandler(_Base): exc, httpx.RemoteProtocolError ) and "peer closed connection without sending complete message body" in str(exc) + @staticmethod+ def _is_timeout_exception(exc: Exception) -> bool:+ return isinstance(exc, httpx.ReadTimeout)+ def _log_tls_info(self, response: httpx.Response, request: Request) -> None: network_stream: AsyncNetworkStream = response.extensions["network_stream"] extra_ssl_object = network_stream.get_extra_info("ssl_object")scrapy/core/downloader/handlers/http11.pyindex d00836fa7..f4f09127b 100644@@ -533,10 +533,21 @@ class _ScrapyAgent: d2: Deferred[_ResultT] = d.addCallback(self._cb_bodyready, request) d3: Deferred[Response] = d2.addCallback(self._cb_bodydone, url) # check download timeout- self._timeout_cl = reactor.callLater(timeout, d3.cancel)+ self._timeout_cl = reactor.callLater(timeout, self._cb_timed_out, d3, request) d3.addBoth(self._cb_timeout, request, url, timeout) return d3 + def _cb_timed_out(self, d3: Deferred[Response], request: Request) -> None:+ fail_on_dataloss = request.meta.get(+ "download_fail_on_dataloss", self._fail_on_dataloss+ )+ if not fail_on_dataloss and self._txresponse is not None:+ # let the response be delivered as a partial one instead of+ # cancelling the request outright+ self._txresponse._transport.stopProducing()+ else:+ d3.cancel()+ def _cb_timeout(self, result: _T, request: Request, url: str, timeout: float) -> _T: if self._timeout_cl.active(): self._timeout_cl.cancel()@@ -546,6 +557,9 @@ class _ScrapyAgent: if self._txresponse: self._txresponse._transport.stopProducing() + if not isinstance(result, Failure):+ return result+ raise DownloadTimeoutError(f"Getting {url} took longer than {timeout} seconds.") def _cb_latency(self, result: _T, request: Request, start_time: float) -> _T:tests/test_downloader_handler_aioftp.pyindex bf98920f7..610c76964 100644@@ -232,10 +232,11 @@ async def test_connection_lost() -> None: def _fake_ftp(- retr_reply: bytes, data: bytes = b""+ retr_reply: bytes | None, data: bytes = b"" ) -> Callable[[asyncio.StreamReader, asyncio.StreamWriter], Any]: """Return a minimal FTP server connection handler that sends *data*, if- any, and then answers RETR with *retr_reply*."""+ any, and then answers RETR with *retr_reply*, or hangs if *retr_reply* is+ ``None``.""" async def handle( reader: asyncio.StreamReader, writer: asyncio.StreamWriter@@ -267,8 +268,12 @@ def _fake_ftp( if data: writer.write(b"150 Sending\r\n") w.write(data)+ if retr_reply is None:+ await reader.read() w.close() data_server.close()+ if retr_reply is None:+ break writer.write(retr_reply) await writer.drain() writer.close()@@ -286,6 +291,7 @@ async def _serve( yield f"ftp://127.0.0.1:{port}/file.txt" finally: server.close()+ await server.wait_closed() @coroutine_test@@ -315,6 +321,19 @@ async def test_dataloss_allowed() -> None: assert response.flags == ["dataloss"] +@coroutine_test+async def test_timeout_partial_response() -> None:+ async with _serve(_fake_ftp(None, data=b"partial")) as url:+ response = await _download(+ Request(+ url,+ meta={"download_timeout": 0.5, "download_fail_on_dataloss": False},+ )+ )+ assert response.body == b"partial"+ assert response.flags == ["dataloss"]++ @coroutine_test async def test_local_filename(ftp_server: MockFTPServer, tmp_path: Path) -> None: local_filename = str(tmp_path / "file.txt").encode()tests/test_downloader_handler_aiohttp.pyindex 72b0811d5..02ef5fe4f 100644@@ -28,6 +28,8 @@ pytestmark = pytest.mark.only_asyncio class AiohttpDownloadHandlerMixin:+ handler_supports_partial_response_on_timeout = True+ @property def download_handler_cls(self) -> type[DownloadHandlerProtocol]: return AiohttpDownloadHandlertests/test_downloader_handler_httpx.pyindex 1fc90f33f..72d2ce834 100644@@ -62,6 +62,8 @@ else: class HttpxDownloadHandlerMixin:+ handler_supports_partial_response_on_timeout = True+ @property def download_handler_cls(self) -> type[DownloadHandlerProtocol]: return HttpxDownloadHandlertests/test_downloader_handler_twisted_http11.pyindex 2c08e6eb0..9e58e3105 100644@@ -56,6 +56,8 @@ pytestmark = pytest.mark.requires_reactor # HTTP11DownloadHandler requires a re class HTTP11DownloadHandlerMixin:+ handler_supports_partial_response_on_timeout = True+ @property def download_handler_cls(self) -> type[DownloadHandlerProtocol]: return HTTP11DownloadHandlertests/utils/bases/download_handlers_http.pyindex a5be39934..985a88276 100644@@ -70,6 +70,9 @@ class TestHttpBase(ABC): http2: bool = False # whether the handler supports per-request bindaddress handler_supports_bindaddress_meta: bool = True+ # whether the handler can return the body received so far, instead of+ # erroring out, when a request times out mid-response+ handler_supports_partial_response_on_timeout: bool = False # whether the handler sends a request Content-Length header as is, instead # of building its own (https://github.com/scrapy/scrapy/issues/4919) handler_supports_custom_content_length: bool = True@@ -435,6 +438,25 @@ class TestHttpBase(ABC): with pytest.raises(DownloadTimeoutError): await maybe_deferred_to_future(d) + @coroutine_test+ async def test_timeout_download_from_spider_server_hangs_partial_response(+ self, mockserver: MockServer+ ) -> None:+ if not self.handler_supports_partial_response_on_timeout:+ pytest.skip("This handler doesn't support partial responses on timeout")+ # client connects, server sends headers and some body bytes but hangs;+ # with dataloss failures disabled the body received so far is+ # returned instead of raising a timeout error+ meta = {"download_timeout": 0.5, "download_fail_on_dataloss": False}+ request = Request(+ mockserver.url("/hang-after-headers", is_secure=self.is_secure), meta=meta+ )+ async with self.get_dh() as download_handler:+ response = await download_handler.download_request(request)+ assert response.status == 200+ assert response.body == b"some bytes"+ assert response.flags == ["dataloss"]+ @pytest.mark.parametrize("send_header", [True, False]) @coroutine_test async def test_host_header(self, send_header: bool, mockserver: MockServer) -> None: