Backtestsscrapyrun-20261025-211112#7774
Document urlparse_cached()?
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches make the identical, correct fix at the root: they add `.. autofunction:: scrapy.utils.httpobj.urlparse_cached` to the existing 'Other functions related to requests' section of request-response.rst, and both independent tests pass on both. The difference is in the docstring quality and tests. Patch A's docstring explains what is returned, that the cache is per-object (same ParseResult instance returned), that weak references mean objects are not kept alive, and includes a short worked example showing `.path` and `.netloc`, which directly mirrors the media-pipeline usage cited in the issue; the doctest values are correct. Patch B's docstring is also accurate and adds a useful 'prefer this over urlparse(x.url)' hint, but is shorter and has no example. Patch A additionally adds focused tests covering Response objects (previously only Request was tested) and the weak-reference non-retention property it now documents; these are non-tautological and exercise behavior the docstring claims, which is a reasonable accompaniment to a docs change rather than churn. The '1 line removed' in A's test file is just the import line being widened to include Response. Neither patch adds risk. On craft and tests A edges out B; correctness and scope are equal.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
2m 53s
Wall-clock time in the container
Full test suite
No new failures
162 passed · 0 skipped · 163 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
50vs 13 merged
Changed lines, ours vs merged
The issue
View on GitHubIt seems useful, is used in some other related projects and used in a code sample on https://docs.scrapy.org/en/latest/topics/media-pipeline.html#scrapy.pipelines.files.FilesPipeline.file_path but not documented.
The two patches
docs/topics/request-response.rstindex 7468f1b64..b83a04032 100644@@ -342,6 +342,8 @@ Other functions related to requests .. autofunction:: scrapy.utils.request.request_from_dict +.. autofunction:: scrapy.utils.httpobj.urlparse_cached+ .. _topics-request-response-ref-request-callback-arguments: scrapy/utils/httpobj.pyindex 58b4539bf..7745cea4d 100644@@ -16,8 +16,24 @@ _urlparse_cache: WeakKeyDictionary[Request | Response, ParseResult] = ( def urlparse_cached(request_or_response: Request | Response) -> ParseResult:- """Return urlparse.urlparse caching the result, where the argument can be a- Request or Response object+ """Return the result of :func:`urllib.parse.urlparse` on the URL of+ *request_or_response*, a :class:`~scrapy.Request` or+ :class:`~scrapy.http.Response` object.++ The result is cached per object, so subsequent calls with the same object+ return the same :class:`~urllib.parse.ParseResult` instance without+ parsing the URL again. The cache holds weak references, so it does not+ keep requests or responses alive.++ For example:++ >>> from scrapy import Request+ >>> from scrapy.utils.httpobj import urlparse_cached+ >>> request = Request("https://example.com/a/b/foo.png?x=1")+ >>> urlparse_cached(request).path+ '/a/b/foo.png'+ >>> urlparse_cached(request).netloc+ 'example.com' """ if request_or_response not in _urlparse_cache: _urlparse_cache[request_or_response] = urlparse(request_or_response.url)tests/test_utils_httpobj.pyindex 0eb330461..1d940ca68 100644@@ -1,7 +1,9 @@+import weakref from urllib.parse import urlparse -from scrapy.http import Request+from scrapy.http import Request, Response from scrapy.utils.httpobj import urlparse_cached+from scrapy.utils.python import garbage_collect def test_urlparse_cached():@@ -18,3 +20,27 @@ def test_urlparse_cached(): assert req1a is req1b assert req1a is not req2 assert req1b is not req2+++def test_urlparse_cached_response():+ url = "https://example.com/a/b/foo.png?x=1"+ response1 = Response(url)+ response2 = Response(url)+ resp1a = urlparse_cached(response1)+ resp1b = urlparse_cached(response1)+ resp2 = urlparse_cached(response2)++ assert resp1a == urlparse(url)+ assert resp1a.path == "/a/b/foo.png"+ assert resp1a is resp1b+ assert resp1a is not resp2+ assert resp1a == resp2+++def test_urlparse_cached_does_not_keep_objects_alive():+ request = Request("http://www.example.com/index.html")+ urlparse_cached(request)+ ref = weakref.ref(request)+ del request+ garbage_collect()+ assert ref() is None
docs/topics/request-response.rstindex 7468f1b64..b83a04032 100644@@ -342,6 +342,8 @@ Other functions related to requests .. autofunction:: scrapy.utils.request.request_from_dict +.. autofunction:: scrapy.utils.httpobj.urlparse_cached+ .. _topics-request-response-ref-request-callback-arguments: scrapy/utils/httpobj.pyindex 58b4539bf..5965e1f21 100644@@ -16,8 +16,15 @@ _urlparse_cache: WeakKeyDictionary[Request | Response, ParseResult] = ( def urlparse_cached(request_or_response: Request | Response) -> ParseResult:- """Return urlparse.urlparse caching the result, where the argument can be a- Request or Response object+ """Return the result of parsing the URL of *request_or_response*, a+ :class:`~scrapy.Request` or :class:`~scrapy.http.Response` object, with+ :func:`urllib.parse.urlparse`.++ The result is cached, using a :class:`weakref.WeakKeyDictionary` keyed on+ *request_or_response*, so that the URL of a given object is parsed only+ once. Prefer this function over calling :func:`urllib.parse.urlparse` on+ ``request_or_response.url`` directly when the same URL may be parsed more+ than once. """ if request_or_response not in _urlparse_cache: _urlparse_cache[request_or_response] = urlparse(request_or_response.url)