Backtestsscrapyrun-20261025-211112#3585
request.meta['download_slot'] is not documented
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches add a `.. reqmeta:: download_slot` target so the existing list entry resolves, and both independent tests pass on each. Patch A puts the documentation exactly where the issue points: a dedicated download_slot section in the Request.meta special keys part of request-response.rst, explaining what the key holds, how the downloader assigns and populates it, which settings it interacts with, and an example of setting it. It adds small cross-links from the autothrottle and DOWNLOAD_SLOTS docs, and ships a test file that fails on base and passes after (the skip condition only triggers when the docs directory is absent, which does not apply to the scenario). Patch B places the target in settings.rst under DOWNLOAD_SLOTS, so a reader of the special keys list is bounced to the settings page rather than finding the key documented in the section the issue names; the explanation is adequate but less complete than A's. More importantly, B bundles large unrelated churn: documenting roughly ten other meta keys, reordering the list, and a source change in scrapy/http/cookies.py that removes support for the `is_unverifiable` meta key and deletes its test. That is a backward-incompatible behavior change with no connection to the issue, adding risk and obscuring the fix, and B adds no test for the actual reported problem. On correctness both resolve the issue, but A is more precisely targeted, better tested, and free of risky scope creep.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
4m 9s
Wall-clock time in the container
Full test suite
No new failures
21 passed · 0 skipped · 21 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
95vs 106 merged
Changed lines, ours vs merged
The issue
View on GitHubrequest.meta['download_slot'] is not documented in https://doc.scrapy.org/en/latest/topics/request-response.html#request-meta-special-keys.
The two patches
docs/topics/autothrottle.rstindex 33289545c..0002ee30d 100644@@ -24,7 +24,8 @@ How it works Scrapy allows defining the concurrency and delay of different download slots, e.g. through the :setting:`DOWNLOAD_SLOTS` setting. By default requests are assigned to slots based on their URL domain, although it is possible to-customize the download slot of any request.+customize the download slot of any request through the :reqmeta:`download_slot`+request meta key. The AutoThrottle extension adjusts the delay of each download slot dynamically, to make your spider send :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` concurrentdocs/topics/request-response.rstindex 7344292a2..6d8c95951 100644@@ -913,6 +913,31 @@ started, i.e. HTTP message sent over the network. This meta key only becomes available when the response has been downloaded. While most other meta keys are used to control Scrapy behavior, this one is supposed to be read-only. +.. reqmeta:: download_slot++download_slot+-------------++The key (a string) of the downloader slot to which the request is assigned.++Requests that share a download slot share its concurrency and delay limits,+i.e. :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, :setting:`DOWNLOAD_DELAY` and+:setting:`RANDOMIZE_DOWNLOAD_DELAY`, which can be customized per slot with the+:setting:`DOWNLOAD_SLOTS` setting. The :ref:`AutoThrottle extension+<topics-autothrottle>` also adjusts delays per download slot.++If this meta key is not set, or is set to ``None``, the downloader assigns the+request to a slot based on the hostname of the request URL, and sets this meta+key to the resulting slot key.++Set this meta key to group requests into a custom download slot, for example,+to apply the same concurrency and delay limits to requests sent to different+domains, or different limits to requests sent to the same domain:++.. code-block:: python++ yield scrapy.Request(url, meta={"download_slot": "slow-api"})+ .. reqmeta:: download_fail_on_dataloss download_fail_on_datalossdocs/topics/settings.rstindex f7f5bf8ae..b5890091b 100644@@ -1092,7 +1092,9 @@ DOWNLOAD_SLOTS Default: ``{}`` -Allows to define concurrency/delay parameters on per slot (domain) basis:+Allows to define concurrency/delay parameters on per slot (domain) basis.+Requests are assigned to slots based on their URL domain by default, or based+on their :reqmeta:`download_slot` request meta key: .. code-block:: python tests/test_docs_reqmeta.pynew file mode 100644index 000000000..add33457e@@ -0,0 +1,63 @@+"""Check that every Request.meta special key referenced in the docs is+documented."""++from __future__ import annotations++import re+from pathlib import Path++import pytest++ROOT = Path(__file__).parent.parent+DOCS_DIR = ROOT / "docs"+SOURCE_DIR = ROOT / "scrapy"++REFERENCE_RE = re.compile(r":reqmeta:`([^`<]+?)(?:\s*<([^>]+)>)?`")+DEFINITION_RE = re.compile(r"^\s*\.\. reqmeta:: (\S+)\s*$", re.MULTILINE)++pytestmark = pytest.mark.skipif(+ not DOCS_DIR.is_dir(), reason="The docs directory is not available"+)+++def _doc_sources() -> list[Path]:+ return [*DOCS_DIR.rglob("*.rst"), *SOURCE_DIR.rglob("*.py")]+++def _defined_reqmeta_keys() -> set[str]:+ keys: set[str] = set()+ for path in _doc_sources():+ keys.update(DEFINITION_RE.findall(path.read_text(encoding="utf-8")))+ return keys+++def _special_keys_list() -> list[str]:+ text = (DOCS_DIR / "topics" / "request-response.rst").read_text(encoding="utf-8")+ section = text.split("Request.meta special keys", 1)[1]+ return [+ match.group(2) or match.group(1)+ for match in REFERENCE_RE.finditer(section.split("\n.. reqmeta::", 1)[0])+ ]+++def test_download_slot_is_documented():+ assert "download_slot" in _defined_reqmeta_keys()+ assert "download_slot" in _special_keys_list()+++def test_special_keys_list_is_documented():+ missing = sorted(set(_special_keys_list()) - _defined_reqmeta_keys())+ assert not missing, f"Request.meta special keys without documentation: {missing}"+++def test_all_reqmeta_references_are_documented():+ defined = _defined_reqmeta_keys()+ missing: list[str] = []+ for path in _doc_sources():+ text = path.read_text(encoding="utf-8")+ for match in REFERENCE_RE.finditer(text):+ key = (match.group(2) or match.group(1)).strip()+ if key not in defined:+ line = text.count("\n", 0, match.start()) + 1+ missing.append(f"{path.relative_to(ROOT)}:{line}: {key}")+ assert not missing, "Undocumented :reqmeta: references:\n" + "\n".join(missing)
docs/topics/downloader-middleware.rstindex 0417ce9e1..c45ec0487 100644@@ -827,6 +827,19 @@ redirect. For example, :class:`RedirectMiddleware` indicates the triggering response status code as an integer, while :class:`MetaRefreshMiddleware` always uses the ``'meta refresh'`` string as reason. +.. reqmeta:: redirect_times++The number of redirects that a request has gone through so far can be found in+the ``redirect_times`` :attr:`Request.meta <scrapy.Request.meta>` key.++.. reqmeta:: redirect_ttl++The number of redirects that a request may still go through is tracked in the+``redirect_ttl`` :attr:`Request.meta <scrapy.Request.meta>` key, which is+initialized from :setting:`REDIRECT_MAX_TIMES` and decreased on every redirect.+Set it on a request to allow fewer redirects for that request;+:setting:`REDIRECT_MAX_TIMES` still applies as an upper bound.+ The :class:`RedirectMiddleware` can be configured through the following settings (see the settings documentation for more info): @@ -984,6 +997,18 @@ RetryMiddleware If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key set to True, the request will be ignored by this middleware. +.. reqmeta:: retry_times++The number of times that a request has been retried so far can be found in the+``retry_times`` :attr:`Request.meta <scrapy.Request.meta>` key. Since it counts+against :setting:`RETRY_TIMES`, copying it into a follow-up request lowers the+number of retries allowed for that request.++.. reqmeta:: priority_adjust++The ``priority_adjust`` :attr:`Request.meta <scrapy.Request.meta>` key+overrides :setting:`RETRY_PRIORITY_ADJUST` for a request.+ To retry requests from a spider callback, you can use the :func:`get_retry_request` function: @@ -1088,6 +1113,8 @@ Adjust retry request priority relative to original request: - a positive priority adjust means higher priority. - **a negative priority adjust (default) means lower priority.** +See also: :reqmeta:`priority_adjust`.+ .. _topics-dlmw-robots: docs/topics/request-response.rstindex 7344292a2..4a6d19ce2 100644@@ -161,7 +161,7 @@ Request objects A common use of request metadata is to define request-specific parameters for Scrapy components (extensions, middlewares, etc.). For- example, if you set ``dont_retry`` to ``True``,+ example, if you set :reqmeta:`dont_retry` to ``True``, :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` will never retry that request, even if it fails. See :ref:`topics-request-meta`. @@ -169,15 +169,15 @@ Request objects example, to keep request state information relevant to your component. For example, :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses the- ``retry_times`` metadata key to keep track of how many times a request- has been retried so far.+ :reqmeta:`retry_times` metadata key to keep track of how many times a+ request has been retried so far. Copying all the metadata of a previous request into a new, follow-up request in a spider callback is a bad practice, because request metadata may include metadata set by Scrapy components that is not meant to be copied into other requests. For example, copying the- ``retry_times`` metadata key into follow-up requests can lower the- amount of retries allowed for those follow-up requests.+ :reqmeta:`retry_times` metadata key into follow-up requests can lower+ the amount of retries allowed for those follow-up requests. You should only copy all request metadata from one request to another if the new request is meant to replace the old request, as is often the@@ -828,6 +828,7 @@ Those are: * :reqmeta:`autothrottle_dont_adjust_delay` * :reqmeta:`bindaddress` * :reqmeta:`cookiejar`+* :reqmeta:`depth` * :reqmeta:`dont_cache` * :reqmeta:`dont_merge_cookies` * :reqmeta:`dont_obey_robotstxt`@@ -837,10 +838,12 @@ Those are: * :reqmeta:`download_latency` * :reqmeta:`download_maxsize` * :reqmeta:`download_slot`-* :reqmeta:`download_warnsize` * :reqmeta:`download_timeout`-* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)-* ``ftp_user`` (See :setting:`FTP_USER` for more info)+* :reqmeta:`download_warnsize`+* :reqmeta:`ftp_local_filename`+* :reqmeta:`ftp_passive`+* :reqmeta:`ftp_password`+* :reqmeta:`ftp_user` * :reqmeta:`give_up_log_level` * :reqmeta:`handle_httpstatus_all` * :reqmeta:`handle_httpstatus_list`@@ -848,13 +851,22 @@ Those are: * :reqmeta:`http_pass` * :reqmeta:`http_user` * :reqmeta:`is_start_request`+* :reqmeta:`link_text` * :reqmeta:`max_retry_times`+* :reqmeta:`priority_adjust` * :reqmeta:`proxy` * :reqmeta:`redirect_reasons`+* :reqmeta:`redirect_times`+* :reqmeta:`redirect_ttl` * :reqmeta:`redirect_urls` * :reqmeta:`referrer_policy`+* :reqmeta:`retry_times`+* :reqmeta:`rule` * :reqmeta:`verbatim_url` +Scrapy components also use meta keys whose name starts with an underscore, such+as ``_auth_proxy``. Those are internal, and may change or disappear at any time.+ .. reqmeta:: bindaddress bindaddress@@ -921,6 +933,15 @@ download_fail_on_dataloss Whether or not to fail on broken responses. See: :setting:`DOWNLOAD_FAIL_ON_DATALOSS`. +.. reqmeta:: ftp_local_filename++ftp_local_filename+------------------++Path, as :class:`bytes`, of the file where to write the response body of an+``ftp://`` request. If set, :attr:`Response.body <scrapy.http.Response.body>`+holds this path instead of the file contents.+ .. reqmeta:: give_up_log_level give_up_log_leveldocs/topics/settings.rstindex f7f5bf8ae..aa4208607 100644@@ -1085,6 +1085,7 @@ handler (without replacement), place this in your ``settings.py``: :ref:`security-local-resources` +.. reqmeta:: download_slot .. setting:: DOWNLOAD_SLOTS DOWNLOAD_SLOTS@@ -1109,6 +1110,14 @@ Allows to define concurrency/delay parameters on per slot (domain) basis: - :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency`` - :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay`` +Requests are assigned to a slot based on their URL domain. To assign a request+to a specific slot instead, set the name of the slot as the ``download_slot``+:attr:`Request.meta <scrapy.Request.meta>` key. Once a request is assigned to a+slot, that key holds the name of the slot.++Since that key is kept on redirects, a redirected request stays in the slot of+the request it comes from, even when it points to a different domain.+ .. setting:: DOWNLOAD_TIMEOUT @@ -1438,6 +1447,7 @@ non-default value in :ref:`per-spider settings <spider-settings>`. .. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`. +.. reqmeta:: ftp_passive .. setting:: FTP_PASSIVE_MODE FTP_PASSIVE_MODE@@ -1445,7 +1455,8 @@ FTP_PASSIVE_MODE Default: ``True`` -Whether or not to use passive mode when initiating FTP transfers.+Whether or not to use passive mode when initiating FTP transfers, unless there+is an ``"ftp_passive"`` key in ``Request`` meta. .. note:: docs/topics/spider-middleware.rstindex 219eff206..cb5608ac6 100644@@ -233,6 +233,8 @@ DepthMiddleware .. module:: scrapy.spidermiddlewares.depth :synopsis: Depth Spider Middleware +.. reqmeta:: depth+ .. autoclass:: DepthMiddleware HttpErrorMiddlewaredocs/topics/spiders.rstindex a932dc72f..ae79ec3ee 100644@@ -491,6 +491,15 @@ CrawlSpider described below. If multiple rules match the same link, the first one will be used, according to the order they're defined in this attribute. + .. reqmeta:: rule++ Requests generated from :attr:`rules` carry the index of the matching rule+ within :attr:`rules` in their ``rule``+ :attr:`Request.meta <scrapy.Request.meta>` key. :class:`CrawlSpider` needs+ that key to dispatch the response to the right rule, so copying it into a+ request generated by a different rule sends the response to the wrong+ callback.+ This spider also exposes an overridable method: .. method:: parse_start_url(response, **kwargs)@@ -504,6 +513,8 @@ CrawlSpider Crawling rules ~~~~~~~~~~~~~~ +.. reqmeta:: link_text+ .. autoclass:: Rule ``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object whichscrapy/http/cookies.pyindex 555d930e6..8008ca9d2 100644@@ -154,14 +154,11 @@ class WrappedRequest: return urlparse_cached(self.request).scheme def is_unverifiable(self) -> bool:- """Unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965.-- It defaults to False. An unverifiable request is one whose URL the user did not have the- option to approve. For example, if the request is for an image in an- HTML document, and the user had no option to approve the automatic- fetching of the image, this should be true.+ """Return ``False``, as Scrapy does not track whether the user had the+ option to approve the URL of a request, which is what makes a request+ unverifiable as defined by :rfc:`2965`. """- return cast("bool", self.request.meta.get("is_unverifiable", False))+ return False @property def full_url(self) -> str:tests/test_http_cookies.pyindex 5d4d5a2bd..8840172a5 100644@@ -79,11 +79,6 @@ class TestWrappedRequest: assert not self.wrapped.is_unverifiable() assert not self.wrapped.unverifiable - def test_is_unverifiable2(self):- self.request.meta["is_unverifiable"] = True- assert self.wrapped.is_unverifiable()- assert self.wrapped.unverifiable- def test_get_origin_req_host(self): assert self.wrapped.origin_req_host == "www.example.com"