Backtestsscrapyrun-20261025-211112#1478

Issue #1478 Merged fix PR #8197 Run 25 Oct 2026 · run-20261025-211112 scrapy/scrapy

Adding support to FTP file listing

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Both patches make a trailing-slash FTP URL perform a listing, and both pass all four independent tests. The differences are in what the listing is and how edges are handled.

Correctness: Patch A issues the FTP LIST command (via twisted's FTPFileListProtocol) and returns a JSON array of structured entries including filetype, size, and name. That matches the issue's wording ('calls FTP list') and actually enables the stated goal of scanning repositories, since a crawler needs to know which entries are directories to recurse. Patch B uses NLST, returning bare names one per line, which gives no way to distinguish files from subdirectories. Patch A also preserves the existing error semantics: a nonexistent directory raises CommandFailed and maps to a 404 like a nonexistent file does. Patch B's own test codifies that a nonexistent directory returns 200 with an empty body, indistinguishable from an empty directory, which is a regression in error reporting relative to the handler's existing conventions. Patch A also keeps ftp_local_filename working for listings and sets the response class to JsonResponse only when the body is the listing itself. Patch B does handle an empty path (ftp://host with no slash) where A would still call retrieveFile(''), a small point in B's favor.

Tests: Patch A adds four tests covering root listing (names, sizes, filetypes, headers), subdirectory listing with a quoted space in the path, nonexistent directory -> 404, and the local-filename path, and extends both fixture trees with a subdirectory so the dir/file distinction is exercised. Patch B adds two tests: root listing names only, and the nonexistent-directory-returns-200-empty behavior, which locks in the weaker semantics.

Scope/craft: Both update the docs. Patch A also updates the module docstring that describes the handler's response shape; B leaves it stale. A's local import inside the helper is slightly unusual but harmless. A's reliance on Unix ls-style LIST parsing is a portability caveat, but NLST's lack of type information is the bigger practical gap for the issue's goal.

Overall A resolves the issue more completely, with better error handling and substantially stronger tests.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

4m 3s

Wall-clock time in the container

Full test suite

No new failures

23 passed · 0 skipped · 23 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

107vs 28 merged

Changed lines, ours vs merged

Make possible to scan FTP repositories. If the path of the request ends in / in a FTP URL, calls FTP list.

Pull Request already open in https://github.com/scrapy/scrapy/pull/941

I hope the pull request gets at least reviewed.

The two patches

Our patch+104 −3
docs/topics/download-handlers.rstindex 4209385f0..7770be4e4 100644@@ -445,6 +445,14 @@ FTPDownloadHandler  This handler supports ``ftp://host/path`` FTP URIs. +If the URI path ends with a slash (e.g. ``ftp://host/some/dir/``), the+directory is listed instead of downloaded, and the response is a+:class:`~scrapy.http.JsonResponse` whose body is a JSON array with one object+per directory entry, with the ``filetype``, ``perms``, ``nlinks``, ``owner``,+``group``, ``size``, ``date``, ``filename`` and ``linktarget`` keys, as parsed+by :class:`twisted.protocols.ftp.FTPFileListProtocol`. Use+``response.json()`` to read it.+ It's implemented using :mod:`twisted.protocols.ftp`.  .. _s3-handler:scrapy/core/downloader/handlers/ftp.pyindex 05bd0ee5d..c42c4d72f 100644@@ -26,10 +26,19 @@ mapping if want a different behaviour than default. In case of status 200 request, response.headers will come with two keys:     'Local Filename' - with the value of the local filename if given     'Size' - with size of the downloaded data++If the URL path ends with a slash (e.g. ``ftp://host/some/dir/``), the directory+is listed (FTP LIST command) instead of retrieving a file. The response body is+then a JSON array with one object per directory entry, as parsed by+twisted.protocols.ftp.FTPFileListProtocol (keys: filetype, perms, nlinks, owner,+group, size, date, filename, linktarget), and the response is a JsonResponse+(unless ftp_local_filename is given, in which case the JSON is saved to that+file). """  from __future__ import annotations +import json import re from io import BytesIO from pathlib import Path@@ -109,9 +118,13 @@ class FTPDownloadHandler(BaseDownloadHandler):             creator.connectTCP(parsed_url.hostname, parsed_url.port or 21)         )         filepath = unquote(parsed_url.path)+        is_listing = filepath.endswith("/")         protocol = ReceivedDataProtocol(request.meta.get("ftp_local_filename"))         try:-            await maybe_deferred_to_future(client.retrieveFile(filepath, protocol))+            if is_listing:+                await self._list_directory(client, filepath, protocol)+            else:+                await maybe_deferred_to_future(client.retrieveFile(filepath, protocol))         except CommandFailed as e:             message = str(e)             # Twisted only raises CommandFailed for a reply whose numeric code@@ -128,5 +141,20 @@ class FTPDownloadHandler(BaseDownloadHandler):             client.transport.loseConnection()         headers = {"Local Filename": protocol.filename or b"", "Size": protocol.size}         body = protocol.filename or protocol.body.read()-        respcls = responsetypes.from_args(url=request.url, body=body)+        if is_listing and not protocol.filename:+            respcls = responsetypes.from_mimetype("application/json")+        else:+            respcls = responsetypes.from_args(url=request.url, body=body)         return respcls(url=request.url, status=200, body=body, headers=headers)++    @staticmethod+    async def _list_directory(+        client: FTPClient, path: str, protocol: ReceivedDataProtocol+    ) -> None:+        """List *path* on the server and write the entries to *protocol* as a+        JSON array."""+        from twisted.protocols.ftp import FTPFileListProtocol++        list_protocol = FTPFileListProtocol()+        await maybe_deferred_to_future(client.list(path, list_protocol))+        protocol.dataReceived(json.dumps(list_protocol.files).encode())tests/test_downloader_handler_twisted_ftp.pyindex fb75a618b..61489ecf2 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import json import os from abc import ABC, abstractmethod from pathlib import Path@@ -15,7 +16,7 @@ from scrapy import Spider from scrapy.core.downloader.handlers.ftp import FTPDownloadHandler from scrapy.crawler import Crawler from scrapy.exceptions import DownloadFailedError, NotConfigured-from scrapy.http import HtmlResponse, Request, Response+from scrapy.http import HtmlResponse, JsonResponse, Request, Response from scrapy.http.response.text import TextResponse from scrapy.utils.defer import deferred_f_from_coro_f, maybe_deferred_to_future from scrapy.utils.misc import build_from_crawler@@ -148,6 +149,66 @@ class TestFTPBase(ABC):         r = await dh.download_request(request)         assert type(r) is response_class +    @deferred_f_from_coro_f+    async def test_ftp_list_directory(+        self, server_url: str, dh: FTPDownloadHandler+    ) -> None:+        request = Request(url=server_url, meta=self.req_meta)+        r = await dh.download_request(request)+        assert r.status == 200+        assert isinstance(r, JsonResponse)+        entries = {entry["filename"]: entry for entry in r.json()}+        assert set(entries) == {filename for filename, _ in self.test_files} | {+            "sub dir"+        }+        assert entries["file.txt"]["size"] == 17+        assert entries["file.txt"]["filetype"] == "-"+        assert entries["sub dir"]["filetype"] == "d"+        assert r.headers[b"Local Filename"] == b""+        assert r.headers[b"Size"] == str(len(r.body)).encode()++    @deferred_f_from_coro_f+    async def test_ftp_list_subdirectory(+        self, server_url: str, dh: FTPDownloadHandler+    ) -> None:+        request = Request(url=server_url + "sub%20dir/", meta=self.req_meta)+        r = await dh.download_request(request)+        assert r.status == 200+        assert isinstance(r, JsonResponse)+        assert [entry["filename"] for entry in r.json()] == ["nested.txt"]+        assert r.json()[0]["size"] == 6++    @deferred_f_from_coro_f+    async def test_ftp_list_nonexistent_directory(+        self, server_url: str, dh: FTPDownloadHandler+    ) -> None:+        request = Request(url=server_url + "nonexistent/", meta=self.req_meta)+        r = await dh.download_request(request)+        assert r.status == 404++    @deferred_f_from_coro_f+    async def test_ftp_list_local_filename(+        self, server_url: str, dh: FTPDownloadHandler+    ) -> None:+        f, local_fname = mkstemp()+        fname_bytes = to_bytes(local_fname)+        local_path = Path(local_fname)+        os.close(f)+        meta = {"ftp_local_filename": fname_bytes}+        meta.update(self.req_meta)+        request = Request(url=server_url + "sub%20dir/", meta=meta)+        r = await dh.download_request(request)+        assert r.status == 200+        assert r.body == fname_bytes+        assert not isinstance(r, JsonResponse)+        listing = json.loads(local_path.read_bytes())+        assert [entry["filename"] for entry in listing] == ["nested.txt"]+        assert r.headers == {+            b"Local Filename": [fname_bytes],+            b"Size": [str(local_path.stat().st_size).encode()],+        }+        local_path.unlink()+  class TestFTP(TestFTPBase):     def _create_files(self, root: Path) -> None:@@ -155,6 +216,8 @@ class TestFTP(TestFTPBase):         userdir.mkdir()         for filename, content in self.test_files:             (userdir / filename).write_bytes(content)+        (userdir / "sub dir").mkdir()+        (userdir / "sub dir" / "nested.txt").write_bytes(b"nested")      def _get_factory(self, root: Path) -> FTPFactory:         from twisted.protocols.ftp import FTPFactory, FTPRealm@@ -187,6 +250,8 @@ class TestAnonymousFTP(TestFTPBase):     def _create_files(self, root: Path) -> None:         for filename, content in self.test_files:             (root / filename).write_bytes(content)+        (root / "sub dir").mkdir()+        (root / "sub dir" / "nested.txt").write_bytes(b"nested")      def _get_factory(self, tmp_path: Path) -> FTPFactory:         from twisted.protocols.ftp import FTPFactory, FTPRealm 
Merged fix · PR #8197+27 −1
docs/topics/download-handlers.rstindex 4209385f0..b3ab386a7 100644@@ -447,6 +447,10 @@ This handler supports ``ftp://host/path`` FTP URIs.  It's implemented using :mod:`twisted.protocols.ftp`. +A path ending in ``/`` is listed instead of downloaded. The response+body is the plain-text output of the FTP ``NLST`` command, one file or+directory name per line.+ .. _s3-handler:  S3DownloadHandlerscrapy/core/downloader/handlers/ftp.pyindex 05bd0ee5d..327193a29 100644@@ -109,9 +109,13 @@ class FTPDownloadHandler(BaseDownloadHandler):             creator.connectTCP(parsed_url.hostname, parsed_url.port or 21)         )         filepath = unquote(parsed_url.path)+        is_listing = filepath == "" or filepath.endswith("/")         protocol = ReceivedDataProtocol(request.meta.get("ftp_local_filename"))         try:-            await maybe_deferred_to_future(client.retrieveFile(filepath, protocol))+            if is_listing:+                await maybe_deferred_to_future(client.nlst(filepath, protocol))+            else:+                await maybe_deferred_to_future(client.retrieveFile(filepath, protocol))         except CommandFailed as e:             message = str(e)             # Twisted only raises CommandFailed for a reply whose numeric codetests/test_downloader_handler_twisted_ftp.pyindex fb75a618b..bea4d44a5 100644@@ -109,6 +109,24 @@ class TestFTPBase(ABC):         assert r.status == 404         assert r.body == b"['550 nonexistent.txt: No such file or directory.']" +    @deferred_f_from_coro_f+    async def test_ftp_list(self, server_url: str, dh: FTPDownloadHandler) -> None:+        request = Request(url=server_url, meta=self.req_meta)+        r = await dh.download_request(request)+        assert isinstance(r, TextResponse)+        assert r.status == 200+        assert set(r.text.splitlines()) == {filename for filename, _ in self.test_files}++    @deferred_f_from_coro_f+    async def test_ftp_list_nonexistent(+        self, server_url: str, dh: FTPDownloadHandler+    ) -> None:+        request = Request(url=server_url + "nonexistent/", meta=self.req_meta)+        r = await dh.download_request(request)+        assert isinstance(r, TextResponse)+        assert r.status == 200+        assert r.text == ""+     @deferred_f_from_coro_f     async def test_ftp_local_filename(         self, server_url: str, dh: FTPDownloadHandler