Backtestsscrapyrun-20261025-211112#8186
Fix timezone handling in FilesPipeline stores
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches make the same correct S3/GCS fix (call `.timestamp()` on the tz-aware datetime instead of `time.mktime(dt.timetuple())`, which reinterpreted UTC wall-clock as local time), and both replace the FTP `float(MDTM)` nonsense with real `YYYYMMDDHHMMSS[.fff]` UTC parsing. The decisive difference is the FTP fractional-seconds edge. RFC 3659 allows an arbitrary number of fraction digits (`1*DIGIT`); Patch A's `_mdtm_to_timestamp` splits on '.' and parses the fraction with `float(f"0.{fraction}")`, so any length works. Patch B uses `strptime` with `%f`, which only accepts 1-6 digits; a longer fraction raises `ValueError`, which `stat_file`'s broad except swallows into `{}` (hence the independent test's `KeyError: 'last_modified'`), silently forcing a re-download. The independent tests confirm: A passes all 9, B fails `test_mdtm_reply_with_long_fraction`. On tests, A's `non_utc_local_timezone` fixture (TZ=+05:45 via tzset, skipped only where tzset is unavailable, i.e. Windows) makes the S3/GCS tests genuinely fail before the fix and asserts hard UNIX constants; its FTP tests cover plain and fractional MDTM against constants and tighten the real-server test to `approx(time.time())`. B's approach of feeding a +09:37-offset datetime is also a valid regression check and is portable, but its FTP test omits the >6-digit case its implementation breaks on. Scope is equivalent; craft slightly favors A's documented helper with doctest examples over inline conditional format selection. A is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
15m 12s
Wall-clock time in the container
Full test suite
No new failures
54 passed · 20 skipped · 74 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
77vs 60 merged
Changed lines, ours vs merged
The issue
View on GitHubFound in https://github.com/scrapy/scrapy/pull/8182#discussion_r4053447020
Various FilesPipeline stores produce the last_modifed value as the UNIX timestamp and some or all of them may do it incorrectly, depending on what are the values returned by S3 and GCS and what is in FTP MDTM. Their tests may also hide these problems and may need to be improved.
The two patches
scrapy/pipelines/files.pyindex 8e49f52e9..b2799b51a 100644@@ -15,6 +15,7 @@ import time import warnings from collections import defaultdict from contextlib import suppress+from datetime import datetime, timezone from ftplib import FTP from io import BytesIO from pathlib import Path@@ -103,6 +104,21 @@ class FilesStoreProtocol(Protocol): ) -> StatInfo | Deferred[StatInfo]: ... +def _mdtm_to_timestamp(value: str) -> float:+ """Convert the time-val of an FTP ``MDTM`` reply (``YYYYMMDDHHMMSS`` with+ an optional fractional part, always in UTC, see :rfc:`3659#section-2.3`)+ to a UNIX timestamp.++ >>> _mdtm_to_timestamp("20191201000000")+ 1575158400.0+ >>> _mdtm_to_timestamp("20191201000000.25")+ 1575158400.25+ """+ whole, _, fraction = value.partition(".")+ dt = datetime.strptime(whole, "%Y%m%d%H%M%S").replace(tzinfo=timezone.utc)+ return dt.timestamp() + (float(f"0.{fraction}") if fraction else 0.0)++ class FSFilesStore: def __init__(self, basedir: str | PathLike[str]): basedir = _to_string(basedir)@@ -201,8 +217,8 @@ class S3FilesStore: @staticmethod def _onsuccess(boto_key: dict[str, Any]) -> StatInfo: checksum = boto_key["ETag"].strip('"')- last_modified = boto_key["LastModified"]- modified_stamp = time.mktime(last_modified.timetuple())+ # botocore returns a timezone-aware (UTC) datetime+ modified_stamp = boto_key["LastModified"].timestamp() return {"checksum": checksum, "last_modified": modified_stamp} def stat_file(@@ -326,7 +342,8 @@ class GCSFilesStore: def _onsuccess(blob: Any) -> StatInfo: if blob: checksum = base64.b64decode(blob.md5_hash).hex()- last_modified = time.mktime(blob.updated.timetuple())+ # blob.updated is a timezone-aware (UTC) datetime+ last_modified = blob.updated.timestamp() return {"checksum": checksum, "last_modified": last_modified} return {} @@ -420,7 +437,9 @@ class FTPFilesStore: if self.USE_ACTIVE_MODE: ftp.set_pasv(False) file_path = f"{self.basedir}/{path}"- last_modified = float(ftp.voidcmd(f"MDTM {file_path}")[4:].strip())+ last_modified = _mdtm_to_timestamp(+ ftp.voidcmd(f"MDTM {file_path}")[4:].strip()+ ) m = hashlib.md5() # noqa: S324 ftp.retrbinary(f"RETR {file_path}", m.update) return {"last_modified": last_modified, "checksum": m.hexdigest()}tests/test_pipeline_files.pyindex c1d9874cb..9ec9348f5 100644@@ -852,6 +852,25 @@ class TestFSFilesStore: assert store.stat_file("full/filename", DUMMY_SPIDER_INFO) == {} +UTC_2019_12_01_TIMESTAMP = 1575158400.0 # 2019-12-01T00:00:00Z+++@pytest.fixture+def non_utc_local_timezone(monkeypatch: pytest.MonkeyPatch):+ """Use a local timezone far from UTC, so that code that wrongly interprets+ UTC times as local times produces wrong results."""+ if not hasattr(time, "tzset"):+ pytest.skip("Changing the local timezone requires time.tzset()")+ # UTC+05:45 as a POSIX TZ string, which works without a tz database+ monkeypatch.setenv("TZ", "<+0545>-05:45")+ time.tzset()+ try:+ yield+ finally:+ monkeypatch.undo()+ time.tzset()++ @pytest.mark.requires_botocore class TestS3FilesStore: @inline_callbacks_test@@ -947,14 +966,15 @@ class TestS3FilesStore: ): store._headers_to_botocore_kwargs({"X-Custom": "value"}) + @pytest.mark.usefixtures("non_utc_local_timezone") @inline_callbacks_test def test_stat(self): bucket = "mybucket" key = "export.csv" uri = f"s3://{bucket}/{key}" checksum = "3187896a9657a28163abb31667df64c8"- # S3FilesStore needs to be fixed to emit tz-aware datetimes- last_modified = datetime(2019, 12, 1) # noqa: DTZ001+ # botocore returns tz-aware UTC datetimes+ last_modified = datetime(2019, 12, 1, tzinfo=timezone.utc) store = S3FilesStore(uri) from botocore.stub import Stubber # noqa: PLC0415@@ -975,7 +995,7 @@ class TestS3FilesStore: file_stats = yield store.stat_file("", info=DUMMY_SPIDER_INFO) assert file_stats == { "checksum": checksum,- "last_modified": last_modified.timestamp(),+ "last_modified": UTC_2019_12_01_TIMESTAMP, } stub.assert_no_pending_responses()@@ -1105,6 +1125,7 @@ class TestGCSFilesStore: ) assert blob.metadata == {} + @pytest.mark.usefixtures("non_utc_local_timezone") @coroutine_test async def test_stat(self) -> None: store, bucket, blob = self.build_gcs_files_store()@@ -1119,7 +1140,7 @@ class TestGCSFilesStore: bucket.get_blob.assert_called_once_with("my_prefix/full/filename") assert stat == { "checksum": checksum,- "last_modified": time.mktime(updated.timetuple()),+ "last_modified": UTC_2019_12_01_TIMESTAMP, } @coroutine_test@@ -1167,7 +1188,7 @@ class TestFTPFileStore: path, buf, info=DUMMY_SPIDER_INFO, meta=meta, headers=None ) stat = yield store.stat_file(path, info=DUMMY_SPIDER_INFO)- assert "last_modified" in stat+ assert stat["last_modified"] == pytest.approx(time.time(), abs=60) assert "checksum" in stat assert stat["checksum"] == "d113d66b2ec7258724a268bd88eef6b6" path = f"{store.basedir}/{path}"@@ -1194,6 +1215,25 @@ class TestFTPFileStore: stat = yield store.stat_file(path, info=DUMMY_SPIDER_INFO) assert stat["checksum"] == "ff1575649a39a27c13faa0d37c84bab3" + @pytest.mark.usefixtures("non_utc_local_timezone")+ @pytest.mark.parametrize(+ ("mdtm_reply", "expected"),+ [+ ("213 20191201000000", UTC_2019_12_01_TIMESTAMP),+ ("213 20191201000000.5", UTC_2019_12_01_TIMESTAMP + 0.5),+ ],+ )+ @inline_callbacks_test+ def test_stat_mdtm(self, mdtm_reply: str, expected: float):+ store = FTPFilesStore("ftp://example.com:21/files")+ ftp = MagicMock()+ ftp.__enter__.return_value = ftp+ ftp.voidcmd.return_value = mdtm_reply+ with mock.patch("scrapy.pipelines.files.FTP", return_value=ftp):+ stat = yield store.stat_file("full/filename", info=DUMMY_SPIDER_INFO)+ ftp.voidcmd.assert_called_once_with("MDTM /files/full/filename")+ assert stat["last_modified"] == expected+ def test_wrong_uri_scheme(self): with pytest.raises( ValueError,
scrapy/pipelines/files.pyindex 8e49f52e9..c5f8ef9f9 100644@@ -15,6 +15,7 @@ import time import warnings from collections import defaultdict from contextlib import suppress+from datetime import datetime, timezone from ftplib import FTP from io import BytesIO from pathlib import Path@@ -202,7 +203,7 @@ class S3FilesStore: def _onsuccess(boto_key: dict[str, Any]) -> StatInfo: checksum = boto_key["ETag"].strip('"') last_modified = boto_key["LastModified"]- modified_stamp = time.mktime(last_modified.timetuple())+ modified_stamp = last_modified.timestamp() return {"checksum": checksum, "last_modified": modified_stamp} def stat_file(@@ -326,7 +327,7 @@ class GCSFilesStore: def _onsuccess(blob: Any) -> StatInfo: if blob: checksum = base64.b64decode(blob.md5_hash).hex()- last_modified = time.mktime(blob.updated.timetuple())+ last_modified = blob.updated.timestamp() return {"checksum": checksum, "last_modified": last_modified} return {} @@ -420,7 +421,13 @@ class FTPFilesStore: if self.USE_ACTIVE_MODE: ftp.set_pasv(False) file_path = f"{self.basedir}/{path}"- last_modified = float(ftp.voidcmd(f"MDTM {file_path}")[4:].strip())+ modified = ftp.voidcmd(f"MDTM {file_path}")[4:].strip()+ time_format = "%Y%m%d%H%M%S.%f" if "." in modified else "%Y%m%d%H%M%S"+ last_modified = (+ datetime.strptime(modified, time_format)+ .replace(tzinfo=timezone.utc)+ .timestamp()+ ) m = hashlib.md5() # noqa: S324 ftp.retrbinary(f"RETR {file_path}", m.update) return {"last_modified": last_modified, "checksum": m.hexdigest()}tests/test_pipeline_files.pyindex c1d9874cb..2d08fdde7 100644@@ -6,7 +6,7 @@ import random import re import time from abc import ABC, abstractmethod-from datetime import datetime, timezone+from datetime import datetime, timedelta, timezone from ftplib import FTP from io import BytesIO from pathlib import Path@@ -953,8 +953,12 @@ class TestS3FilesStore: key = "export.csv" uri = f"s3://{bucket}/{key}" checksum = "3187896a9657a28163abb31667df64c8"- # S3FilesStore needs to be fixed to emit tz-aware datetimes- last_modified = datetime(2019, 12, 1) # noqa: DTZ001+ last_modified = datetime(+ 2019,+ 12,+ 1,+ tzinfo=timezone(timedelta(hours=9, minutes=37)),+ ) store = S3FilesStore(uri) from botocore.stub import Stubber # noqa: PLC0415@@ -1110,7 +1114,12 @@ class TestGCSFilesStore: store, bucket, blob = self.build_gcs_files_store() checksum = "cdcda85605e46d0af6110752770dce3c" blob.md5_hash = base64.b64encode(bytes.fromhex(checksum)).decode()- updated = datetime(2019, 12, 1, tzinfo=timezone.utc)+ updated = datetime(+ 2019,+ 12,+ 1,+ tzinfo=timezone(timedelta(hours=9, minutes=37)),+ ) blob.updated = updated bucket.get_blob.return_value = blob stat = await maybe_deferred_to_future(@@ -1119,7 +1128,7 @@ class TestGCSFilesStore: bucket.get_blob.assert_called_once_with("my_prefix/full/filename") assert stat == { "checksum": checksum,- "last_modified": time.mktime(updated.timetuple()),+ "last_modified": updated.timestamp(), } @coroutine_test@@ -1168,6 +1177,7 @@ class TestFTPFileStore: ) stat = yield store.stat_file(path, info=DUMMY_SPIDER_INFO) assert "last_modified" in stat+ assert stat["last_modified"] == pytest.approx(time.time(), abs=60) assert "checksum" in stat assert stat["checksum"] == "d113d66b2ec7258724a268bd88eef6b6" path = f"{store.basedir}/{path}"@@ -1181,6 +1191,33 @@ class TestFTPFileStore: ) assert data == content + @pytest.mark.parametrize(+ ("response", "expected"),+ [+ ("213 20191201010203", 1575162123.0),+ ("213 20191201010203.123456", 1575162123.123456),+ ],+ )+ @inline_callbacks_test+ def test_stat_timestamp(+ self,+ monkeypatch: pytest.MonkeyPatch,+ response: str,+ expected: float,+ ):+ ftp = MagicMock()+ ftp.__enter__.return_value = ftp+ ftp.voidcmd.return_value = response+ ftp.retrbinary.side_effect = lambda command, callback: callback(b"data")+ monkeypatch.setattr(files, "FTP", lambda: ftp)+ monkeypatch.setattr(FTPFilesStore, "FTP_USERNAME", "anonymous")+ monkeypatch.setattr(FTPFilesStore, "FTP_PASSWORD", "guest")++ store = FTPFilesStore("ftp://example.com:21/")+ stat = yield store.stat_file("full/filename", info=DUMMY_SPIDER_INFO)++ assert stat["last_modified"] == expected+ @inline_callbacks_test def test_persist_active_mode(self, monkeypatch: pytest.MonkeyPatch): data = b"active mode"