Backtestsscrapyrun-20261025-211112#5166
HTTP cache should be for project but spider
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the root cause the same way: an opt-in setting that drops the spider-name partition from both built-in storages, and both pass every independent test. The difference is scope and test coverage. Patch A adds a single boolean HTTPCACHE_SHARED, touches exactly the two path/db-name computations, and its new tests live in the shared StorageTestMixin and DummyPolicyTestMixin, so they run against both FilesystemCacheStorage and DbmCacheStorage, cover the per-spider default, the shared case, isolation between shared and non-shared caches, and an end-to-end middleware hit with the 'cached' flag and httpcache/hit stat. Its only extra change is a harmless mkdir(exist_ok=True) cleanup. Patch B introduces a string enum HTTPCACHE_SCOPE and, beyond the issue's ask, a whole new 'domain' partitioning mode (hostname quoting, filesystem-only, with an IPv6 case that is skipped under a w3lib condition), plus an unrelated removal of the 'If empty, the HTTP cache will be disabled' sentence from HTTPCACHE_DIR docs. The value validation with a clear ValueError is a nice craft touch, and the enum is extensible, but the extra mode adds surface area and risk that the issue did not request, and B's tests stop at the storage layer with no middleware-level check. On correctness they are equal; on tests A is somewhat broader for the behaviour actually asked for; on scope A is clearly tighter. A is the better fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
4m 44s
Wall-clock time in the container
Full test suite
No new failures
343 passed · 3 skipped · 346 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
89vs 157 merged
Changed lines, ours vs merged
The issue
View on GitHubAfter setting HTTPCACHE_ENABLED = True I find that the file storage is separated by spider name. So the web page is still reCrawled in another spider. That makes this feature a dup of duplicate url filter. I think the cache should be valid for all spiders in project.
The two patches
docs/topics/downloader-middleware.rstindex 6bc9df4ab..823c2dd19 100644@@ -527,6 +527,11 @@ Filesystem storage backend (default) /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7 + where ``example.com`` is the spider name. If :setting:`HTTPCACHE_SHARED`+ is ``True``, the spider name directory is omitted::++ /path/to/cache/dir/72/72811f648e718090f041317756c03adb0ada46c7+ .. _httpcache-storage-dbm: DBM storage backend@@ -539,6 +544,9 @@ DBM storage backend By default, it uses the :mod:`dbm`, but you can change it with the :setting:`HTTPCACHE_DBM_MODULE` setting. + The database is stored in the cache directory as ``<spider name>.db``, or+ as ``shared.db`` if :setting:`HTTPCACHE_SHARED` is ``True``.+ .. _httpcache-storage-custom: Writing your own storage backend@@ -648,6 +656,25 @@ project data dir. For more info see: :ref:`topics-project-structure`. Scrapy project source code. Do not point :setting:`HTTPCACHE_DIR` to a path that untrusted parties can write to. +.. setting:: HTTPCACHE_SHARED++HTTPCACHE_SHARED+^^^^^^^^^^^^^^^^++.. versionadded:: VERSION++Default: ``False``++By default, the built-in storage backends keep a separate cache for each+spider, keyed by the spider name. If ``True``, all spiders that use the same+:setting:`HTTPCACHE_DIR` share a single cache, so a response cached by one+spider is served from the cache to any other spider that sends a request with+the same fingerprint.++.. note:: Some DBM implementations (e.g. :mod:`dbm.gnu`) do not allow the same+ database to be opened more than once at a time, so running several spiders+ concurrently with a shared :class:`DbmCacheStorage` cache may fail.+ .. setting:: HTTPCACHE_IGNORE_HTTP_CODES HTTPCACHE_IGNORE_HTTP_CODESdocs/topics/request-response.rstindex ca4b71842..3c6d47c3d 100644@@ -446,7 +446,8 @@ The following built-in Scrapy components have such restrictions: :setting:`HTTPCACHE_DIR` also apply. Inside :setting:`HTTPCACHE_DIR`, the following directory structure is created: - - :attr:`.Spider.name`+ - :attr:`.Spider.name` (omitted if :setting:`HTTPCACHE_SHARED` is+ ``True``) - first byte of a request fingerprint as hexadecimal scrapy/extensions/httpcache.pyindex 171cea4b4..6772e12cc 100644@@ -250,10 +250,12 @@ class DbmCacheStorage: self.cachedir: str = data_path(settings["HTTPCACHE_DIR"], createdir=True) self.expiration_secs: int = settings.getint("HTTPCACHE_EXPIRATION_SECS") self.dbmodule: ModuleType = import_module(settings["HTTPCACHE_DBM_MODULE"])+ self.shared: bool = settings.getbool("HTTPCACHE_SHARED") self.db: Any = None # the real type is private def open_spider(self, spider: Spider) -> None:- dbpath = Path(self.cachedir, f"{spider.name}.db")+ dbname = "shared" if self.shared else spider.name+ dbpath = Path(self.cachedir, f"{dbname}.db") self.db = self.dbmodule.open(str(dbpath), "c") logger.debug(@@ -306,6 +308,7 @@ class FilesystemCacheStorage: self.cachedir: str = data_path(settings["HTTPCACHE_DIR"]) self.expiration_secs: int = settings.getint("HTTPCACHE_EXPIRATION_SECS") self.use_gzip: bool = settings.getbool("HTTPCACHE_GZIP")+ self.shared: bool = settings.getbool("HTTPCACHE_SHARED") # https://github.com/python/mypy/issues/10740 self._open: Callable[ Concatenate[str | os.PathLike[str], str, ...], IO[bytes]@@ -353,8 +356,7 @@ class FilesystemCacheStorage: ) -> None: """Store the given response in the cache.""" rpath = Path(self._get_request_path(spider, request))- if not rpath.exists():- rpath.mkdir(parents=True)+ rpath.mkdir(parents=True, exist_ok=True) metadata = { "url": request.url, "method": request.method,@@ -384,6 +386,8 @@ class FilesystemCacheStorage: def _get_request_path(self, spider: Spider, request: Request) -> str: key = self._fingerprinter.fingerprint(request).hex()+ if self.shared:+ return str(Path(self.cachedir, key[0:2], key)) return str(Path(self.cachedir, spider.name, key[0:2], key)) def _read_meta(self, spider: Spider, request: Request) -> dict[str, Any] | None:scrapy/settings/default_settings.pyindex 4cb4a9add..df970d7d9 100644@@ -128,6 +128,7 @@ __all__ = [ "HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS", "HTTPCACHE_IGNORE_SCHEMES", "HTTPCACHE_POLICY",+ "HTTPCACHE_SHARED", "HTTPCACHE_STORAGE", "HTTPERROR_ALLOWED_CODES", "HTTPERROR_ALLOW_ALL",@@ -450,6 +451,7 @@ HTTPCACHE_IGNORE_MISSING = False HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS: list[str] = [] HTTPCACHE_IGNORE_SCHEMES = ["file"] HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy"+HTTPCACHE_SHARED = False HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage" HTTPERROR_ALLOW_ALL = Falsetests/test_downloadermiddleware_httpcache.pyindex ec054426a..6c906622a 100644@@ -77,10 +77,12 @@ class TestBase: return settings @contextmanager- def _get_crawler(self, **new_settings: Any) -> Generator[Crawler]:+ def _get_crawler(+ self, spider_name: str = "example.com", **new_settings: Any+ ) -> Generator[Crawler]: settings = self._get_settings(**new_settings) crawler = get_crawler(Spider, settings)- crawler.spider = crawler._create_spider("example.com")+ crawler.spider = crawler._create_spider(spider_name) crawler.stats.open_spider() try: yield crawler@@ -88,13 +90,15 @@ class TestBase: crawler.stats.close_spider() @contextmanager- def _storage(self, **new_settings: Any):- with self._middleware(**new_settings) as mw:+ def _storage(self, spider_name: str = "example.com", **new_settings: Any):+ with self._middleware(spider_name, **new_settings) as mw: yield mw.storage, mw.crawler @contextmanager- def _middleware(self, **new_settings: Any) -> Generator[HttpCacheMiddleware]:- with self._get_crawler(**new_settings) as crawler:+ def _middleware(+ self, spider_name: str = "example.com", **new_settings: Any+ ) -> Generator[HttpCacheMiddleware]:+ with self._get_crawler(spider_name, **new_settings) as crawler: assert crawler.spider mw = build_from_crawler(HttpCacheMiddleware, crawler) mw.spider_opened(crawler.spider)@@ -231,6 +235,27 @@ class StorageTestMixin(TestBase): assert isinstance(cached_response, HtmlResponse) self.assertEqualResponse(self.response, cached_response) + def test_storage_per_spider_by_default(self):+ with self._storage("spider1") as (storage, crawler):+ storage.store_response(crawler.spider, self.request, self.response)+ with self._storage("spider2") as (storage, crawler):+ assert storage.retrieve_response(crawler.spider, self.request) is None+ with self._storage("spider1") as (storage, crawler):+ cached_response = storage.retrieve_response(crawler.spider, self.request)+ assert cached_response is not None+ self.assertEqualResponse(self.response, cached_response)++ def test_storage_shared(self):+ with self._storage("spider1", HTTPCACHE_SHARED=True) as (storage, crawler):+ storage.store_response(crawler.spider, self.request, self.response)+ with self._storage("spider2", HTTPCACHE_SHARED=True) as (storage, crawler):+ cached_response = storage.retrieve_response(crawler.spider, self.request)+ assert cached_response is not None+ self.assertEqualResponse(self.response, cached_response)+ # A non-shared cache does not see the shared entries.+ with self._storage("spider2") as (storage, crawler):+ assert storage.retrieve_response(crawler.spider, self.request) is None+ class PolicyTestMixin(TestBase): """Mixin containing policy-specific test methods."""@@ -271,6 +296,16 @@ class PolicyTestMixin(TestBase): class DummyPolicyTestMixin(PolicyTestMixin): """Mixin containing dummy policy specific test methods.""" + def test_middleware_shared(self):+ with self._middleware("spider1", HTTPCACHE_SHARED=True) as mw:+ mw.storage.store_response(mw.crawler.spider, self.request, self.response)+ with self._middleware("spider2", HTTPCACHE_SHARED=True) as mw:+ cached_response = mw.process_request(self.request.copy())+ assert isinstance(cached_response, Response)+ self.assertEqualResponse(self.response, cached_response)+ assert "cached" in cached_response.flags+ assert mw.crawler.stats.get_value("httpcache/hit") == 1+ def test_middleware(self): with self._middleware() as mw: assert mw.process_request(self.request) is None
docs/topics/downloader-middleware.rstindex 6bc9df4ab..b99a7483e 100644@@ -523,9 +523,10 @@ Filesystem storage backend (default) The directory name is made from the request fingerprint (see ``scrapy.utils.request.fingerprint``), and one level of subdirectories is used to avoid creating too many files into the same directory (which is- inefficient in many file systems). An example directory could be::+ inefficient in many file systems). Those directories live inside the+ folder that :setting:`HTTPCACHE_SCOPE` selects, ``my_spider`` below:: - /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7+ /path/to/cache/dir/my_spider/72/72811f648e718090f041317756c03adb0ada46c7 .. _httpcache-storage-dbm: @@ -640,9 +641,40 @@ HTTPCACHE_DIR Default: ``'httpcache'`` -The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP-cache will be disabled. If a relative path is given, is taken relative to the-project data dir. For more info see: :ref:`topics-project-structure`.+The directory to use for storing the (low-level) HTTP cache. If a relative path+is given, is taken relative to the project data dir. For more info see:+:ref:`topics-project-structure`.++.. setting:: HTTPCACHE_SCOPE++HTTPCACHE_SCOPE+^^^^^^^^^^^^^^^++Default: ``'spider'``++.. versionadded:: VERSION++How to partition the cache within :setting:`HTTPCACHE_DIR`:++``'spider'``+ One cache per spider. Spiders never reuse each other's cached responses.++``'none'``+ A single cache, shared by every spider.++``'domain'``+ One cache per request host, shared by every spider. Only+ :class:`~scrapy.extensions.httpcache.FilesystemCacheStorage` supports this+ value; it allows removing the cached responses of a single website by+ deleting its directory.++Changing this setting makes existing cached responses unreachable, since+nothing looks for them in their old location. Delete+:setting:`HTTPCACHE_DIR` to reclaim their disk space.++With :class:`~scrapy.extensions.httpcache.DbmCacheStorage`, ``'none'`` makes+every spider use the same database file, which most DBM implementations open+for exclusive writing. Spiders that share a cache must then run one at a time. .. warning:: Treat the cache directory with the same security care as your Scrapy project source code. Do not point :setting:`HTTPCACHE_DIR` to adocs/topics/request-response.rstindex ca4b71842..b20fe73d3 100644@@ -446,7 +446,9 @@ The following built-in Scrapy components have such restrictions: :setting:`HTTPCACHE_DIR` also apply. Inside :setting:`HTTPCACHE_DIR`, the following directory structure is created: - - :attr:`.Spider.name`+ - the partition that :setting:`HTTPCACHE_SCOPE` selects, i.e.+ :attr:`.Spider.name` or the request host, and no folder at all with+ ``'none'`` - first byte of a request fingerprint as hexadecimal scrapy/extensions/httpcache.pyindex 171cea4b4..f6f92a525 100644@@ -9,6 +9,7 @@ from importlib import import_module from pathlib import Path from time import time from typing import IO, TYPE_CHECKING, Any, Concatenate, cast+from urllib.parse import quote from weakref import WeakKeyDictionary from w3lib.http import headers_dict_to_raw, headers_raw_to_dict@@ -33,6 +34,17 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +def _get_scope(settings: BaseSettings, supported: tuple[str, ...]) -> str:+ scope: str = settings["HTTPCACHE_SCOPE"]+ if scope not in supported:+ supported_repr = ", ".join(repr(value) for value in supported)+ raise ValueError(+ f"Unsupported HTTPCACHE_SCOPE value: {scope!r}. The cache storage "+ f"in use supports the following values: {supported_repr}."+ )+ return scope++ class DummyPolicy: def __init__(self, settings: BaseSettings): self.ignore_schemes: list[str] = settings.getlist("HTTPCACHE_IGNORE_SCHEMES")@@ -251,9 +263,11 @@ class DbmCacheStorage: self.expiration_secs: int = settings.getint("HTTPCACHE_EXPIRATION_SECS") self.dbmodule: ModuleType = import_module(settings["HTTPCACHE_DBM_MODULE"]) self.db: Any = None # the real type is private+ self._scope = _get_scope(settings, ("spider", "none")) def open_spider(self, spider: Spider) -> None:- dbpath = Path(self.cachedir, f"{spider.name}.db")+ name = spider.name if self._scope == "spider" else "httpcache"+ dbpath = Path(self.cachedir, f"{name}.db") self.db = self.dbmodule.open(str(dbpath), "c") logger.debug(@@ -306,6 +320,7 @@ class FilesystemCacheStorage: self.cachedir: str = data_path(settings["HTTPCACHE_DIR"]) self.expiration_secs: int = settings.getint("HTTPCACHE_EXPIRATION_SECS") self.use_gzip: bool = settings.getbool("HTTPCACHE_GZIP")+ self._scope = _get_scope(settings, ("spider", "none", "domain")) # https://github.com/python/mypy/issues/10740 self._open: Callable[ Concatenate[str | os.PathLike[str], str, ...], IO[bytes]@@ -384,7 +399,17 @@ class FilesystemCacheStorage: def _get_request_path(self, spider: Spider, request: Request) -> str: key = self._fingerprinter.fingerprint(request).hex()- return str(Path(self.cachedir, spider.name, key[0:2], key))+ scope_path = self._get_scope_path(spider, request)+ return str(Path(self.cachedir, scope_path, key[0:2], key))++ def _get_scope_path(self, spider: Spider, request: Request) -> str:+ if self._scope == "spider":+ return spider.name+ if self._scope == "domain":+ # quote() covers hostnames that are not valid path components,+ # such as IPv6 addresses.+ return quote(urlparse_cached(request).hostname or "_", safe="")+ return "" def _read_meta(self, spider: Spider, request: Request) -> dict[str, Any] | None: rpath = Path(self._get_request_path(spider, request))scrapy/settings/default_settings.pyindex 4cb4a9add..479b6cc16 100644@@ -128,6 +128,7 @@ __all__ = [ "HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS", "HTTPCACHE_IGNORE_SCHEMES", "HTTPCACHE_POLICY",+ "HTTPCACHE_SCOPE", "HTTPCACHE_STORAGE", "HTTPERROR_ALLOWED_CODES", "HTTPERROR_ALLOW_ALL",@@ -450,6 +451,7 @@ HTTPCACHE_IGNORE_MISSING = False HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS: list[str] = [] HTTPCACHE_IGNORE_SCHEMES = ["file"] HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy"+HTTPCACHE_SCOPE = "spider" HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage" HTTPERROR_ALLOW_ALL = Falsetests/test_downloadermiddleware_httpcache.pyindex ec054426a..d80dd945c 100644@@ -10,6 +10,7 @@ from contextlib import contextmanager from pathlib import Path from typing import TYPE_CHECKING, Any from unittest import mock+from urllib.parse import urlparse import pytest @@ -18,7 +19,7 @@ from scrapy.exceptions import IgnoreRequest from scrapy.extensions.httpcache import DummyPolicy, rfc1123_to_epoch from scrapy.http import HtmlResponse, Request, Response from scrapy.spiders import Spider-from scrapy.utils.misc import build_from_crawler+from scrapy.utils.misc import build_from_crawler, load_object from scrapy.utils.test import get_crawler if TYPE_CHECKING:@@ -867,6 +868,83 @@ class TestFilesystemStorageGzipWithDummyPolicy(TestFilesystemStorageWithDummyPol body_path.write_bytes(body_path.read_bytes()[:-5]) +class ScopeTestMixin(TestBase):+ policy_class = "scrapy.extensions.httpcache.DummyPolicy"++ @contextmanager+ def _spider_storage(self, crawler: Crawler, name: str) -> Generator[Any]:+ spider = crawler._create_spider(name)+ storage = load_object(self.storage_class)(crawler.settings)+ storage.open_spider(spider)+ try:+ yield storage, spider+ finally:+ storage.close_spider(spider)++ def _cross_spider_retrieval(self, scope: str) -> Any:+ # Storages are opened one at a time because a DBM database shared by+ # two spiders does not support concurrent access.+ with self._get_crawler(HTTPCACHE_SCOPE=scope) as crawler:+ with self._spider_storage(crawler, "a") as (storage, spider):+ storage.store_response(spider, self.request, self.response)+ with self._spider_storage(crawler, "b") as (storage, spider):+ return storage.retrieve_response(spider, self.request)++ def test_spider_scope(self):+ assert self._cross_spider_retrieval("spider") is None++ def test_no_scope(self):+ assert self._cross_spider_retrieval("none") is not None++ def test_unknown_scope(self):+ with (+ self._get_crawler(HTTPCACHE_SCOPE="unknown") as crawler,+ pytest.raises(ValueError, match="Unsupported HTTPCACHE_SCOPE"),+ ):+ load_object(self.storage_class)(crawler.settings)+++class TestFilesystemStorageScope(ScopeTestMixin):+ storage_class = "scrapy.extensions.httpcache.FilesystemCacheStorage"++ def test_domain_scope(self):+ assert self._cross_spider_retrieval("domain") is not None++ @pytest.mark.parametrize(+ ("url", "expected"),+ [+ ("http://user:pass@WWW.Example.com:8080/", "www.example.com"),+ pytest.param(+ "http://[::1]/",+ "%3A%3A1",+ marks=pytest.mark.skipif(+ urlparse(Request("http://[::1]/").url).hostname != "::1",+ reason="w3lib strips the brackets of IPv6 hosts",+ ),+ ),+ ("file:///tmp/t.txt", "_"),+ ],+ )+ def test_domain_scope_path(self, url, expected):+ with (+ self._get_crawler(HTTPCACHE_SCOPE="domain") as crawler,+ self._spider_storage(crawler, "a") as (storage, spider),+ ):+ path = Path(storage._get_request_path(spider, Request(url)))+ assert path.relative_to(self.tmpdir).parts[0] == expected+++class TestDbmStorageScope(ScopeTestMixin):+ storage_class = "scrapy.extensions.httpcache.DbmCacheStorage"++ def test_domain_scope_unsupported(self):+ with (+ self._get_crawler(HTTPCACHE_SCOPE="domain") as crawler,+ pytest.raises(ValueError, match="Unsupported HTTPCACHE_SCOPE"),+ ):+ load_object(self.storage_class)(crawler.settings)++ @pytest.mark.parametrize( ("string", "expected"), [